2024-09-21T01:45:45.292 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T01:45:45.323 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T01:45:45.409 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iogen}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913403' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: mds: mds_bal_replicate_threshold: 1 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy mgr-modules: - snap_schedule sha1: 1843221895adccfb3b692060251dd6a0183bc410 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAPMafp1Oqman7acTJtIpvzIoa/jw9NfT7hHZo1Yvs8GjOWmQ2x+elU08ITM7U14l6mAJGvthvfBJIeL7e1Zhds= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDOzLBSMqeDpLKD2gKr8TqE0PcP81izGE1pblcR4skPXU+4Qni2Yd24UPDQxK+4KQEIxiRsBkjI8TgfbWspl1xs= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXCh7nGnG+/W2TXp+pmbxqvizCw+EiGus1i6WgFgyLiIEanehpz8svtTz07KNBzkfLxAt5Qkc9uQEkwcJ6mOao= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T01:45:45.410 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T01:45:45.410 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T01:45:45.410 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T01:45:45.412 INFO:teuthology.task.internal:Checking packages... 2024-09-21T01:45:45.432 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T01:45:45.432 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T01:45:45.432 INFO:teuthology.packaging:ref: None 2024-09-21T01:45:45.433 INFO:teuthology.packaging:tag: None 2024-09-21T01:45:45.433 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:45:45.433 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:45:45.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:45:45.584 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T01:45:45.585 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T01:45:45.591 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T01:45:45.592 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T01:45:45.609 INFO:teuthology.task.internal:Saving configuration 2024-09-21T01:45:45.635 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T01:45:45.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T01:45:45.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 01:40:41.474651', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAPMafp1Oqman7acTJtIpvzIoa/jw9NfT7hHZo1Yvs8GjOWmQ2x+elU08ITM7U14l6mAJGvthvfBJIeL7e1Zhds='} 2024-09-21T01:45:45.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 01:40:41.476708', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDOzLBSMqeDpLKD2gKr8TqE0PcP81izGE1pblcR4skPXU+4Qni2Yd24UPDQxK+4KQEIxiRsBkjI8TgfbWspl1xs='} 2024-09-21T01:45:45.702 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 01:40:41.478431', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXCh7nGnG+/W2TXp+pmbxqvizCw+EiGus1i6WgFgyLiIEanehpz8svtTz07KNBzkfLxAt5Qkc9uQEkwcJ6mOao='} 2024-09-21T01:45:45.703 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T01:45:45.709 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-21T01:45:45.709 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-21T01:45:45.709 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-21T01:45:45.709 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T01:45:45.805 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc9aa50160>, signals=[15]) 2024-09-21T01:45:45.805 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T01:45:45.811 INFO:teuthology.task.internal:Opening connections... 2024-09-21T01:45:45.811 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T01:45:45.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:45:45.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-09-21T01:45:45.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:45:45.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-09-21T01:45:45.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:45:46.037 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T01:45:46.044 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-21T01:45:46.062 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-21T01:45:46.062 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-21T01:45:46.120 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-09-21T01:45:46.120 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-09-21T01:45:46.120 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-09-21T01:45:46.120 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T01:45:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T01:45:46.122 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-21T01:45:46.146 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-09-21T01:45:46.165 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-09-21T01:45:46.165 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T01:45:46.223 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T01:45:46.224 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T01:45:46.224 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-09-21T01:45:46.246 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-09-21T01:45:46.264 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-09-21T01:45:46.264 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-09-21T01:45:46.321 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-09-21T01:45:46.322 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T01:45:46.323 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T01:45:46.323 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T01:45:46.323 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-09-21T01:45:46.345 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T01:45:46.354 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T01:45:46.433 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T01:45:46.433 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-21T01:45:46.436 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-09-21T01:45:46.438 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-09-21T01:45:46.455 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T01:45:46.461 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T01:45:46.461 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-21T01:45:46.493 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-09-21T01:45:46.495 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-09-21T01:45:46.544 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T01:45:46.616 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T01:45:46.630 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T01:45:46.630 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T01:45:46.630 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T01:45:46.631 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T01:45:46.631 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T01:45:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:45:46.648 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-09-21T01:45:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:45:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:45:46.650 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-09-21T01:45:46.650 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-21T01:45:46.705 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-503.el9.x86_64 2024-09-21T01:45:46.705 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.14.0-503.el9.x86_64 2024-09-21T01:45:46.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2024-09-21T01:45:46.707 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-503.el9.x86_64 2024-09-21T01:45:46.708 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-503.el9.x86_64 2024-09-21T01:45:46.708 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.14.0-503.el9.x86_64 2024-09-21T01:45:46.708 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-09-21T01:45:46.708 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-503.el9.x86_64 2024-09-21T01:45:46.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-09-21T01:45:48.879 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-21T01:45:48.976 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-21T01:45:49.313 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T01:45:50.605 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-21T01:45:51.205 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-21T01:45:52.361 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 20 MB 00:02 2024-09-21T01:45:56.673 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.5 MB 00:01 2024-09-21T01:45:56.863 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-21T01:45:58.448 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-21T01:45:59.072 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-21T01:45:59.236 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-21T01:45:59.269 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.5 MB 00:03 2024-09-21T01:45:59.968 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-21T01:46:00.752 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-21T01:46:01.648 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T01:46:04.909 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T01:46:05.424 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-21T01:46:06.899 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T01:46:06.899 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T01:46:07.003 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T01:46:07.012 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:46:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 79 M 2024-09-21T01:46:07.014 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 126 M 2024-09-21T01:46:07.014 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-21T01:46:07.087 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T01:46:07.177 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 309 kB/s | 36 kB 00:00 2024-09-21T01:46:07.296 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T01:46:07.297 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T01:46:07.397 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-09-21T01:46:07.402 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout:Install 4 Packages 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:46:07.403 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 79 M 2024-09-21T01:46:07.404 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 126 M 2024-09-21T01:46:07.404 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-09-21T01:46:07.825 INFO:teuthology.orchestra.run.smithi123.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 112 kB/s | 36 kB 00:00 2024-09-21T01:46:07.852 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-21T01:46:08.452 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 25 MB/s | 35 MB 00:01 2024-09-21T01:46:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 17 MB/s | 29 MB 00:01 2024-09-21T01:46:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:46:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-21T01:46:08.944 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T01:46:08.944 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T01:46:09.065 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-09-21T01:46:09.075 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2024-09-21T01:46:09.076 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:46:09.076 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 79 M 2024-09-21T01:46:09.076 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 126 M 2024-09-21T01:46:09.076 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-09-21T01:46:09.214 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 308 kB/s | 36 kB 00:00 2024-09-21T01:46:09.525 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-21T01:46:09.748 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-09-21T01:46:09.915 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-21T01:46:09.916 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-21T01:46:10.506 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 25 MB/s | 35 MB 00:01 2024-09-21T01:46:10.773 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 18 MB/s | 29 MB 00:01 2024-09-21T01:46:10.773 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:46:10.773 INFO:teuthology.orchestra.run.smithi161.stdout:Total 46 MB/s | 79 MB 00:01 2024-09-21T01:46:10.877 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-21T01:46:10.877 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-21T01:46:11.456 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-09-21T01:46:11.828 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-09-21T01:46:11.828 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-09-21T01:46:12.831 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-09-21T01:46:12.832 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-09-21T01:46:13.431 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-21T01:46:14.081 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:46:14.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:15.253 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-09-21T01:46:15.908 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:46:15.990 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:18.777 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:18.918 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:22.174 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:22.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:46:23.757 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:23.894 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:46:24.781 INFO:teuthology.orchestra.run.smithi123.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 928 kB/s | 16 MB 00:17 2024-09-21T01:46:27.354 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:46:28.738 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:46:35.122 INFO:teuthology.orchestra.run.smithi123.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 1.0 MB/s | 29 MB 00:27 2024-09-21T01:46:36.331 INFO:teuthology.orchestra.run.smithi123.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 1.2 MB/s | 35 MB 00:28 2024-09-21T01:46:36.331 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:46:36.331 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.7 MB/s | 79 MB 00:28 2024-09-21T01:46:37.018 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-09-21T01:46:37.378 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-09-21T01:46:37.378 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-09-21T01:46:38.346 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-09-21T01:46:38.346 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-09-21T01:46:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-09-21T01:46:41.606 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:46:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:44.438 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:46:44.567 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:49.408 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:46:49.517 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:46:54.388 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.043 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.083 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:47:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:47:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:47:07.395 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:47:07.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:47:07.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:47:07.931 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-21T01:47:08.219 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:47:08.220 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-09-21T01:47:08.291 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:08.374 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.375 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.375 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.375 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T01:47:08.375 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T01:47:08.375 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T01:47:08.375 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T01:47:08.375 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:08.434 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.434 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-09-21T01:47:08.467 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-09-21T01:47:08.467 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 82.el9 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8274670 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-09-21T01:47:08.468 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:47:08.469 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T01:47:08.470 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T01:47:08.470 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T01:47:08.564 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:08.654 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.654 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.654 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.654 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T01:47:08.654 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T01:47:08.654 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-21T01:47:08.655 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T01:47:08.655 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:08.713 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:08.713 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.713 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.06 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 82.el9 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Group : Unspecified 2024-09-21T01:47:08.746 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 8274670 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : builder@centos.org 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:47:08.747 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T01:47:08.748 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T01:47:08.748 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T01:47:08.748 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2024-09-21T01:47:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:47:08.748 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T01:47:08.749 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T01:47:08.749 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 5.14.0-508.el9.x86_64 2024-09-21T01:47:08.749 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T01:47:09.175 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-21T01:47:09.360 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-09-21T01:47:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T01:47:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-21T01:47:10.554 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-21T01:47:10.569 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.rqQmAf50o6 2024-09-21T01:47:10.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rqQmAf50o6 2024-09-21T01:47:10.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.rqQmAf50o6 2024-09-21T01:47:10.691 INFO:teuthology.orchestra.run.smithi161.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T01:47:10.708 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-09-21T01:47:10.711 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-09-21T01:47:10.726 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.tHlm1LCGrw 2024-09-21T01:47:10.726 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tHlm1LCGrw 2024-09-21T01:47:10.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.tHlm1LCGrw 2024-09-21T01:47:10.811 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.rqQmAf50o6 is 6KB 2024-09-21T01:47:10.832 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.rqQmAf50o6 2024-09-21T01:47:10.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T01:47:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T01:47:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T01:47:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T01:47:10.917 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T01:47:10.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T01:47:10.933 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.tHlm1LCGrw is 6KB 2024-09-21T01:47:10.981 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.tHlm1LCGrw 2024-09-21T01:47:10.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T01:47:11.058 INFO:teuthology.orchestra.run.smithi161.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T01:47:11.059 INFO:teuthology.orchestra.run.smithi161.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T01:47:11.059 INFO:teuthology.orchestra.run.smithi161.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T01:47:11.059 INFO:teuthology.orchestra.run.smithi161.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T01:47:11.060 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T01:47:11.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-21T01:47:11.209 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-09-21T01:47:32.364 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:32.403 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-09-21T01:47:34.007 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:34.085 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:34.085 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:34.085 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T01:47:34.085 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T01:47:34.085 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T01:47:34.085 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T01:47:34.086 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T01:47:34.086 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:47:34.144 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:47:34.144 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:47:34.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 82.el9 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-09-21T01:47:34.176 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8274670 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T01:47:34.177 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:47:34.178 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T01:47:34.179 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T01:47:34.179 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 5.14.0-508.el9.x86_64 2024-09-21T01:47:34.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T01:47:34.820 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-09-21T01:47:36.130 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T01:47:36.165 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-09-21T01:47:36.168 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-09-21T01:47:36.183 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.Bv9PKye2E6 2024-09-21T01:47:36.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Bv9PKye2E6 2024-09-21T01:47:36.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.Bv9PKye2E6 2024-09-21T01:47:36.387 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.Bv9PKye2E6 is 6KB 2024-09-21T01:47:36.435 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.Bv9PKye2E6 2024-09-21T01:47:36.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T01:47:36.512 INFO:teuthology.orchestra.run.smithi123.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T01:47:36.512 INFO:teuthology.orchestra.run.smithi123.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T01:47:36.512 INFO:teuthology.orchestra.run.smithi123.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T01:47:36.512 INFO:teuthology.orchestra.run.smithi123.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T01:47:36.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T01:47:36.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-09-21T01:47:41.074 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T01:47:41.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:47:41.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:47:41.212 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-21T01:47:41.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:47:41.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:47:59.599 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-09-21T01:47:59.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-21T01:48:06.675 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T01:48:06.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:48:06.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:08.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:48:08.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:08.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:48:08.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:11.667 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-09-21T01:48:23.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:48:23.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:25.199 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-09-21T01:48:34.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:48:34.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:37.263 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-09-21T01:48:42.095 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-09-21T01:48:49.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:48:49.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:57.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:48:57.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:48:57.606 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-09-21T01:48:57.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:48:57.971 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-508.el9.x86_64"... 2024-09-21T01:48:57.971 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-09-21T01:48:57.986 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-508.el9.x86_64 2024-09-21T01:48:57.986 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T01:48:57.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T01:48:57.986 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T01:48:58.987 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-21T01:48:58.988 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T01:48:59.067 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-09-21T01:48:59.092 DEBUG:teuthology.parallel:result is None 2024-09-21T01:49:08.606 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T01:49:20.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:49:20.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:49:20.994 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-21T01:49:21.145 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-09-21T01:49:21.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:49:21.357 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T01:49:21.357 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-21T01:49:21.374 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-508.el9.x86_64 2024-09-21T01:49:21.374 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T01:49:21.374 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T01:49:21.374 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T01:49:21.757 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:49:21.757 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T01:49:21.757 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-09-21T01:49:21.773 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-508.el9.x86_64 2024-09-21T01:49:21.773 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T01:49:21.773 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T01:49:21.773 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T01:49:22.375 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T01:49:22.376 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T01:49:22.459 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-21T01:49:22.481 DEBUG:teuthology.parallel:result is None 2024-09-21T01:49:22.774 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T01:49:22.775 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T01:49:22.850 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-09-21T01:49:22.869 DEBUG:teuthology.parallel:result is None 2024-09-21T01:49:22.870 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T01:49:22.877 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T01:49:22.877 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T01:49:22.879 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T01:49:22.882 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T01:49:22.901 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T01:49:22.910 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T01:49:22.988 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T01:49:22.988 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T01:49:22.991 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T01:49:22.993 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T01:49:23.028 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T01:49:23.034 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T01:49:23.034 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T01:49:23.054 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T01:49:23.063 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T01:49:23.089 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.094 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.096 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.101 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.105 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.107 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:49:23.108 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T01:49:23.116 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T01:49:23.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T01:49:23.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T01:49:23.151 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T01:49:23.179 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T01:49:23.187 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T01:49:23.188 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T01:49:23.217 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T01:49:23.220 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T01:49:23.237 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.369 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:23.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:23.500 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:49:23.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T01:49:23.570 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.663 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:23.744 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:23.786 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:49:23.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T01:49:23.851 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.898 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:49:23.949 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:24.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:49:24.076 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:49:24.076 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T01:49:24.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-21T01:49:24.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-09-21T01:49:24.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-09-21T01:49:24.195 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:49:24.201 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:49:24.264 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:49:24.744 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T01:49:24.758 INFO:teuthology.task.internal:Starting timer... 2024-09-21T01:49:24.758 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T01:49:24.820 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T01:49:24.853 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T01:49:24.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-09-21T01:49:24.927 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-09-21T01:49:24.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-09-21T01:49:24.987 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-09-21T01:49:24.989 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-09-21T01:49:25.049 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-09-21T01:49:25.051 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T01:49:25.052 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-09-21T01:49:25.080 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-09-21T01:49:25.081 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-09-21T01:49:25.108 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-09-21T01:49:25.109 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-09-21T01:49:25.135 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-09-21T01:49:25.136 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-09-21T01:49:25.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:49:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:49:25.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:49:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:49:25.196 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T01:49:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:49:25.226 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T01:49:25.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-09-21T01:49:25.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-09-21T01:49:25.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-09-21T01:49:25.339 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T01:49:25.347 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T01:49:25.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T01:49:25.360 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi123.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T01:53:09.995 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-09-21T01:53:09.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:53:09.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:53:10.075 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-21T01:53:10.121 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-21T01:53:10.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:53:10.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:53:10.195 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-09-21T01:53:10.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-21T01:53:10.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:53:10.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:53:10.345 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-09-21T01:53:10.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-21T01:53:10.422 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T01:53:10.432 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T01:53:10.433 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T01:53:10.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:53:10.436 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T01:53:10.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:53:10.438 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T01:53:10.438 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:53:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T01:53:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T01:53:10.478 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T01:53:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T01:53:10.492 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T01:53:10.502 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-09-21T01:53:10.506 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T01:53:10.513 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-09-21T01:53:10.513 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-09-21T01:53:10.525 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-09-21T01:53:10.527 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T01:53:10.531 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-09-21T01:53:10.542 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T01:53:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T01:53:10.542 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-09-21T01:53:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T01:53:10.557 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T01:53:10.571 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T01:53:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.605 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.623 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:53:10.626 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:53:10.626 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-09-21T01:53:10.627 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.627 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.627 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.627 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:53:10.627 INFO:teuthology.run_tasks:Running task install... 2024-09-21T01:53:10.635 DEBUG:teuthology.task.install:project ceph 2024-09-21T01:53:10.635 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T01:53:10.635 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T01:53:10.636 INFO:teuthology.task.install:Using flavor: default 2024-09-21T01:53:10.641 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T01:53:10.641 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-21T01:53:10.641 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T01:53:10.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:53:10.643 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T01:53:10.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:53:10.644 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T01:53:10.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:53:10.792 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T01:53:10.792 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T01:53:10.802 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T01:53:10.802 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T01:53:10.807 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T01:53:10.807 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T01:53:10.908 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T01:53:10.909 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:53:10.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T01:53:10.918 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T01:53:10.918 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:53:10.918 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T01:53:10.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T01:53:10.920 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:53:10.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T01:53:10.936 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T01:53:10.937 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T01:53:10.944 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T01:53:10.944 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T01:53:10.946 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-21T01:53:10.946 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T01:53:11.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T01:53:11.009 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T01:53:11.011 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T01:53:11.083 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T01:53:11.084 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T01:53:11.085 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T01:53:11.148 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-09-21T01:53:11.148 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-09-21T01:53:11.150 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-09-21T01:53:11.150 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-09-21T01:53:11.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-09-21T01:53:11.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-09-21T01:53:11.379 INFO:teuthology.orchestra.run.smithi161.stdout:45 files removed 2024-09-21T01:53:11.386 INFO:teuthology.orchestra.run.smithi123.stdout:45 files removed 2024-09-21T01:53:11.394 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-09-21T01:53:11.406 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T01:53:11.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T01:53:11.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-21T01:53:11.947 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 330 kB/s | 84 kB 00:00 2024-09-21T01:53:11.956 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 325 kB/s | 84 kB 00:00 2024-09-21T01:53:11.992 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-21T01:53:12.222 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 105 kB/s | 24 kB 00:00 2024-09-21T01:53:12.241 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-21T01:53:12.284 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-21T01:53:12.481 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 8.6 kB/s | 1.9 kB 00:00 2024-09-21T01:53:12.508 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-21T01:53:12.560 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T01:53:12.941 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.3 MB 00:00 2024-09-21T01:53:13.310 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T01:53:14.098 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.3 MB 00:01 2024-09-21T01:53:15.346 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-21T01:53:15.808 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-21T01:53:17.017 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 6.8 MB/s | 20 MB 00:02 2024-09-21T01:53:20.234 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-21T01:53:21.182 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-21T01:53:21.701 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-21T01:53:22.618 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-21T01:53:22.839 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-21T01:53:23.007 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-21T01:53:24.028 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-21T01:53:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-21T01:53:24.948 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-21T01:53:28.294 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-21T01:53:29.617 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T01:53:30.420 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-21T01:53:30.505 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:30.505 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:30.517 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T01:53:30.518 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T01:53:30.519 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T01:53:30.519 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T01:53:30.521 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T01:53:30.523 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T01:53:30.524 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T01:53:30.586 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-09-21T01:53:30.600 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T01:53:30.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T01:53:30.602 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T01:53:30.603 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T01:53:30.604 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T01:53:30.605 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T01:53:30.606 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T01:53:30.607 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T01:53:30.608 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T01:53:30.609 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T01:53:30.610 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T01:53:30.611 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:Install 149 Packages 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 8 Packages 2024-09-21T01:53:30.612 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:53:30.613 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 228 M 2024-09-21T01:53:30.613 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-09-21T01:53:31.087 INFO:teuthology.orchestra.run.smithi161.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 45 kB/s | 6.6 kB 00:00 2024-09-21T01:53:31.263 INFO:teuthology.orchestra.run.smithi161.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 4.9 MB/s | 873 kB 00:00 2024-09-21T01:53:31.338 INFO:teuthology.orchestra.run.smithi161.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 1.9 MB/s | 148 kB 00:00 2024-09-21T01:53:31.430 INFO:teuthology.orchestra.run.smithi161.stdout:(4/157): ceph-base-19.3.0-5120.g18432218.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-21T01:53:31.555 INFO:teuthology.orchestra.run.smithi161.stdout:(5/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 12 MB/s | 1.5 MB 00:00 2024-09-21T01:53:31.605 INFO:teuthology.orchestra.run.smithi161.stdout:(6/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 8.6 MB/s | 2.3 MB 00:00 2024-09-21T01:53:31.856 INFO:teuthology.orchestra.run.smithi161.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 17 MB/s | 5.0 MB 00:00 2024-09-21T01:53:31.913 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:31.914 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:31.928 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T01:53:31.930 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T01:53:31.931 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T01:53:31.932 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T01:53:31.934 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T01:53:31.937 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T01:53:31.938 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T01:53:32.006 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T01:53:32.024 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T01:53:32.025 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T01:53:32.026 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T01:53:32.027 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T01:53:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T01:53:32.029 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T01:53:32.030 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T01:53:32.031 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T01:53:32.032 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T01:53:32.033 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T01:53:32.034 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:Install 149 Packages 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 8 Packages 2024-09-21T01:53:32.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:53:32.036 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 228 M 2024-09-21T01:53:32.036 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-21T01:53:32.064 INFO:teuthology.orchestra.run.smithi161.stdout:(8/157): ceph-common-19.3.0-5120.g18432218.el9. 18 MB/s | 20 MB 00:01 2024-09-21T01:53:32.098 INFO:teuthology.orchestra.run.smithi161.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 762 kB/s | 25 kB 00:00 2024-09-21T01:53:32.532 INFO:teuthology.orchestra.run.smithi161.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 15 MB/s | 10 MB 00:00 2024-09-21T01:53:32.607 INFO:teuthology.orchestra.run.smithi161.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 2.9 MB/s | 226 kB 00:00 2024-09-21T01:53:32.646 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:32.647 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-21T01:53:32.658 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-21T01:53:32.659 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-21T01:53:32.660 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-21T01:53:32.661 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-21T01:53:32.662 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-21T01:53:32.665 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-21T01:53:32.666 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-21T01:53:32.669 INFO:teuthology.orchestra.run.smithi016.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 47 kB/s | 6.6 kB 00:00 2024-09-21T01:53:32.728 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T01:53:32.750 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 226 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell x86_64 2:19.3.0-5120.g18432218.el9 ceph 27 k 2024-09-21T01:53:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 18 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T01:53:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-21T01:53:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T01:53:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T01:53:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T01:53:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T01:53:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T01:53:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T01:53:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T01:53:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T01:53:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:Install 149 Packages 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 8 Packages 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 228 M 2024-09-21T01:53:32.762 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-09-21T01:53:32.782 INFO:teuthology.orchestra.run.smithi161.stdout:(12/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 15 MB/s | 18 MB 00:01 2024-09-21T01:53:32.807 INFO:teuthology.orchestra.run.smithi161.stdout:(13/157): cephfs-shell-19.3.0-5120.g18432218.el 135 kB/s | 27 kB 00:00 2024-09-21T01:53:32.849 INFO:teuthology.orchestra.run.smithi161.stdout:(14/157): libcephfs-devel-19.3.0-5120.g18432218 477 kB/s | 32 kB 00:00 2024-09-21T01:53:32.900 INFO:teuthology.orchestra.run.smithi161.stdout:(15/157): libcephfs2-19.3.0-5120.g18432218.el9. 8.1 MB/s | 766 kB 00:00 2024-09-21T01:53:32.933 INFO:teuthology.orchestra.run.smithi161.stdout:(16/157): libcephsqlite-19.3.0-5120.g18432218.e 1.8 MB/s | 158 kB 00:00 2024-09-21T01:53:32.936 INFO:teuthology.orchestra.run.smithi016.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 3.2 MB/s | 873 kB 00:00 2024-09-21T01:53:32.975 INFO:teuthology.orchestra.run.smithi161.stdout:(17/157): librados-devel-19.3.0-5120.g18432218. 1.6 MB/s | 127 kB 00:00 2024-09-21T01:53:33.000 INFO:teuthology.orchestra.run.smithi161.stdout:(18/157): libradosstriper1-19.3.0-5120.g1843221 7.4 MB/s | 507 kB 00:00 2024-09-21T01:53:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:(3/157): ceph-base-19.3.0-5120.g18432218.el9.x8 11 MB/s | 5.5 MB 00:00 2024-09-21T01:53:33.044 INFO:teuthology.orchestra.run.smithi016.stdout:(4/157): ceph-immutable-object-cache-19.3.0-512 1.3 MB/s | 148 kB 00:00 2024-09-21T01:53:33.059 INFO:teuthology.orchestra.run.smithi161.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 772 kB/s | 45 kB 00:00 2024-09-21T01:53:33.159 INFO:teuthology.orchestra.run.smithi161.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 1.4 MB/s | 148 kB 00:00 2024-09-21T01:53:33.203 INFO:teuthology.orchestra.run.smithi016.stdout:(5/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 12 MB/s | 2.3 MB 00:00 2024-09-21T01:53:33.235 INFO:teuthology.orchestra.run.smithi161.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 2.1 MB/s | 161 kB 00:00 2024-09-21T01:53:33.343 INFO:teuthology.orchestra.run.smithi161.stdout:(22/157): python3-rados-19.3.0-5120.g18432218.e 2.9 MB/s | 321 kB 00:00 2024-09-21T01:53:33.370 INFO:teuthology.orchestra.run.smithi016.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-09-21T01:53:33.396 INFO:teuthology.orchestra.run.smithi123.stdout:(1/157): ceph-19.3.0-5120.g18432218.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-09-21T01:53:33.452 INFO:teuthology.orchestra.run.smithi161.stdout:(23/157): librgw2-19.3.0-5120.g18432218.el9.x86 11 MB/s | 5.3 MB 00:00 2024-09-21T01:53:33.477 INFO:teuthology.orchestra.run.smithi161.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 2.2 MB/s | 307 kB 00:00 2024-09-21T01:53:33.552 INFO:teuthology.orchestra.run.smithi161.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 992 kB/s | 99 kB 00:00 2024-09-21T01:53:33.554 INFO:teuthology.orchestra.run.smithi016.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 14 MB/s | 5.0 MB 00:00 2024-09-21T01:53:33.578 INFO:teuthology.orchestra.run.smithi161.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 861 kB/s | 86 kB 00:00 2024-09-21T01:53:33.903 INFO:teuthology.orchestra.run.smithi161.stdout:(27/157): rbd-mirror-19.3.0-5120.g18432218.el9. 8.9 MB/s | 3.1 MB 00:00 2024-09-21T01:53:33.978 INFO:teuthology.orchestra.run.smithi161.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 457 kB/s | 34 kB 00:00 2024-09-21T01:53:34.070 INFO:teuthology.orchestra.run.smithi161.stdout:(29/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 1.6 MB/s | 154 kB 00:00 2024-09-21T01:53:34.279 INFO:teuthology.orchestra.run.smithi161.stdout:(30/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 243 kB/s | 170 kB 00:00 2024-09-21T01:53:34.321 INFO:teuthology.orchestra.run.smithi016.stdout:(8/157): ceph-radosgw-19.3.0-5120.g18432218.el9 14 MB/s | 10 MB 00:00 2024-09-21T01:53:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 335 kB/s | 25 kB 00:00 2024-09-21T01:53:34.530 INFO:teuthology.orchestra.run.smithi123.stdout:(2/157): ceph-fuse-19.3.0-5120.g18432218.el9.x8 771 kB/s | 873 kB 00:01 2024-09-21T01:53:34.713 INFO:teuthology.orchestra.run.smithi123.stdout:(3/157): ceph-immutable-object-cache-19.3.0-512 809 kB/s | 148 kB 00:00 2024-09-21T01:53:34.763 INFO:teuthology.orchestra.run.smithi016.stdout:(10/157): ceph-common-19.3.0-5120.g18432218.el9 8.9 MB/s | 20 MB 00:02 2024-09-21T01:53:34.822 INFO:teuthology.orchestra.run.smithi016.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.8 MB/s | 226 kB 00:00 2024-09-21T01:53:34.897 INFO:teuthology.orchestra.run.smithi016.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 361 kB/s | 27 kB 00:00 2024-09-21T01:53:34.955 INFO:teuthology.orchestra.run.smithi016.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 544 kB/s | 32 kB 00:00 2024-09-21T01:53:34.988 INFO:teuthology.orchestra.run.smithi161.stdout:(31/157): ceph-test-19.3.0-5120.g18432218.el9.x 14 MB/s | 40 MB 00:02 2024-09-21T01:53:35.081 INFO:teuthology.orchestra.run.smithi016.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 6.0 MB/s | 766 kB 00:00 2024-09-21T01:53:35.138 INFO:teuthology.orchestra.run.smithi161.stdout:(32/157): ceph-mgr-diskprediction-local-19.3.0- 8.6 MB/s | 7.4 MB 00:00 2024-09-21T01:53:35.148 INFO:teuthology.orchestra.run.smithi016.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.3 MB/s | 158 kB 00:00 2024-09-21T01:53:35.163 INFO:teuthology.orchestra.run.smithi161.stdout:(33/157): ceph-mgr-modules-core-19.3.0-5120.g18 1.6 MB/s | 292 kB 00:00 2024-09-21T01:53:35.188 INFO:teuthology.orchestra.run.smithi161.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 988 kB/s | 49 kB 00:00 2024-09-21T01:53:35.207 INFO:teuthology.orchestra.run.smithi016.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 2.1 MB/s | 127 kB 00:00 2024-09-21T01:53:35.222 INFO:teuthology.orchestra.run.smithi161.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 280 kB/s | 16 kB 00:00 2024-09-21T01:53:35.255 INFO:teuthology.orchestra.run.smithi161.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 4.2 MB/s | 285 kB 00:00 2024-09-21T01:53:35.290 INFO:teuthology.orchestra.run.smithi016.stdout:(17/157): libradosstriper1-19.3.0-5120.g1843221 6.0 MB/s | 507 kB 00:00 2024-09-21T01:53:35.331 INFO:teuthology.orchestra.run.smithi161.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 245 kB/s | 18 kB 00:00 2024-09-21T01:53:35.364 INFO:teuthology.orchestra.run.smithi161.stdout:(38/157): cephadm-19.3.0-5120.g18432218.el9.noa 5.4 MB/s | 784 kB 00:00 2024-09-21T01:53:35.515 INFO:teuthology.orchestra.run.smithi161.stdout:(39/157): ceph-mgr-dashboard-19.3.0-5120.g18432 9.8 MB/s | 14 MB 00:01 2024-09-21T01:53:35.540 INFO:teuthology.orchestra.run.smithi161.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 382 kB/s | 80 kB 00:00 2024-09-21T01:53:35.607 INFO:teuthology.orchestra.run.smithi016.stdout:(18/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 8.0 MB/s | 18 MB 00:02 2024-09-21T01:53:35.649 INFO:teuthology.orchestra.run.smithi161.stdout:(41/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-09-21T01:53:35.649 INFO:teuthology.orchestra.run.smithi016.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 1.1 MB/s | 45 kB 00:00 2024-09-21T01:53:35.674 INFO:teuthology.orchestra.run.smithi161.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-21T01:53:35.708 INFO:teuthology.orchestra.run.smithi016.stdout:(20/157): python3-ceph-common-19.3.0-5120.g1843 2.5 MB/s | 148 kB 00:00 2024-09-21T01:53:35.708 INFO:teuthology.orchestra.run.smithi161.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-21T01:53:35.733 INFO:teuthology.orchestra.run.smithi161.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-21T01:53:35.758 INFO:teuthology.orchestra.run.smithi161.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-21T01:53:35.775 INFO:teuthology.orchestra.run.smithi016.stdout:(21/157): python3-cephfs-19.3.0-5120.g18432218. 2.4 MB/s | 161 kB 00:00 2024-09-21T01:53:35.800 INFO:teuthology.orchestra.run.smithi161.stdout:(46/157): python3-cryptography-36.0.1-4.el9.x86 8.1 MB/s | 1.2 MB 00:00 2024-09-21T01:53:35.825 INFO:teuthology.orchestra.run.smithi161.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-09-21T01:53:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:(22/157): librgw2-19.3.0-5120.g18432218.el9.x86 9.4 MB/s | 5.3 MB 00:00 2024-09-21T01:53:35.851 INFO:teuthology.orchestra.run.smithi161.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-21T01:53:35.855 INFO:teuthology.orchestra.run.smithi123.stdout:(4/157): ceph-mds-19.3.0-5120.g18432218.el9.x86 2.0 MB/s | 2.3 MB 00:01 2024-09-21T01:53:35.875 INFO:teuthology.orchestra.run.smithi016.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 3.1 MB/s | 321 kB 00:00 2024-09-21T01:53:35.909 INFO:teuthology.orchestra.run.smithi016.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 5.1 MB/s | 307 kB 00:00 2024-09-21T01:53:35.934 INFO:teuthology.orchestra.run.smithi016.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.7 MB/s | 99 kB 00:00 2024-09-21T01:53:35.947 INFO:teuthology.orchestra.run.smithi123.stdout:(5/157): ceph-base-19.3.0-5120.g18432218.el9.x8 1.9 MB/s | 5.5 MB 00:02 2024-09-21T01:53:35.960 INFO:teuthology.orchestra.run.smithi016.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.7 MB/s | 86 kB 00:00 2024-09-21T01:53:36.010 INFO:teuthology.orchestra.run.smithi016.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-21T01:53:36.018 INFO:teuthology.orchestra.run.smithi161.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 642 kB/s | 107 kB 00:00 2024-09-21T01:53:36.068 INFO:teuthology.orchestra.run.smithi161.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 435 kB/s | 22 kB 00:00 2024-09-21T01:53:36.069 INFO:teuthology.orchestra.run.smithi016.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 586 kB/s | 34 kB 00:00 2024-09-21T01:53:36.118 INFO:teuthology.orchestra.run.smithi161.stdout:(51/157): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-21T01:53:36.144 INFO:teuthology.orchestra.run.smithi016.stdout:(29/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 2.0 MB/s | 154 kB 00:00 2024-09-21T01:53:36.185 INFO:teuthology.orchestra.run.smithi161.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-09-21T01:53:36.194 INFO:teuthology.orchestra.run.smithi016.stdout:(30/157): rbd-mirror-19.3.0-5120.g18432218.el9. 12 MB/s | 3.1 MB 00:00 2024-09-21T01:53:36.210 INFO:teuthology.orchestra.run.smithi161.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-09-21T01:53:36.236 INFO:teuthology.orchestra.run.smithi161.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-09-21T01:53:36.261 INFO:teuthology.orchestra.run.smithi161.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-09-21T01:53:36.286 INFO:teuthology.orchestra.run.smithi161.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-21T01:53:36.320 INFO:teuthology.orchestra.run.smithi161.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.8 MB/s | 171 kB 00:00 2024-09-21T01:53:36.345 INFO:teuthology.orchestra.run.smithi161.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-21T01:53:36.370 INFO:teuthology.orchestra.run.smithi161.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-21T01:53:36.462 INFO:teuthology.orchestra.run.smithi161.stdout:(60/157): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-09-21T01:53:36.489 INFO:teuthology.orchestra.run.smithi123.stdout:(6/157): ceph-mgr-19.3.0-5120.g18432218.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-09-21T01:53:36.496 INFO:teuthology.orchestra.run.smithi161.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-21T01:53:36.513 INFO:teuthology.orchestra.run.smithi161.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 486 kB/s | 24 kB 00:00 2024-09-21T01:53:36.538 INFO:teuthology.orchestra.run.smithi161.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-21T01:53:36.580 INFO:teuthology.orchestra.run.smithi161.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-21T01:53:36.613 INFO:teuthology.orchestra.run.smithi161.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-09-21T01:53:36.630 INFO:teuthology.orchestra.run.smithi161.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-21T01:53:36.786 INFO:teuthology.orchestra.run.smithi016.stdout:(31/157): ceph-test-19.3.0-5120.g18432218.el9.x 17 MB/s | 40 MB 00:02 2024-09-21T01:53:36.837 INFO:teuthology.orchestra.run.smithi016.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 5.7 MB/s | 292 kB 00:00 2024-09-21T01:53:36.954 INFO:teuthology.orchestra.run.smithi016.stdout:(33/157): ceph-mgr-diskprediction-local-19.3.0- 9.7 MB/s | 7.4 MB 00:00 2024-09-21T01:53:36.979 INFO:teuthology.orchestra.run.smithi016.stdout:(34/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 347 kB/s | 49 kB 00:00 2024-09-21T01:53:36.997 INFO:teuthology.orchestra.run.smithi161.stdout:(67/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-21T01:53:37.004 INFO:teuthology.orchestra.run.smithi016.stdout:(35/157): ceph-prometheus-alerts-19.3.0-5120.g1 329 kB/s | 16 kB 00:00 2024-09-21T01:53:37.030 INFO:teuthology.orchestra.run.smithi016.stdout:(36/157): ceph-volume-19.3.0-5120.g18432218.el9 5.6 MB/s | 285 kB 00:00 2024-09-21T01:53:37.031 INFO:teuthology.orchestra.run.smithi123.stdout:(7/157): ceph-mon-19.3.0-5120.g18432218.el9.x86 4.6 MB/s | 5.0 MB 00:01 2024-09-21T01:53:37.080 INFO:teuthology.orchestra.run.smithi016.stdout:(37/157): cephadm-19.3.0-5120.g18432218.el9.noa 10 MB/s | 784 kB 00:00 2024-09-21T01:53:37.097 INFO:teuthology.orchestra.run.smithi016.stdout:(38/157): cephfs-top-19.3.0-5120.g18432218.el9. 275 kB/s | 18 kB 00:00 2024-09-21T01:53:37.156 INFO:teuthology.orchestra.run.smithi161.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-21T01:53:37.281 INFO:teuthology.orchestra.run.smithi161.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 7.1 MB/s | 910 kB 00:00 2024-09-21T01:53:37.314 INFO:teuthology.orchestra.run.smithi016.stdout:(39/157): ceph-mgr-dashboard-19.3.0-5120.g18432 12 MB/s | 14 MB 00:01 2024-09-21T01:53:37.347 INFO:teuthology.orchestra.run.smithi016.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 299 kB/s | 80 kB 00:00 2024-09-21T01:53:37.348 INFO:teuthology.orchestra.run.smithi161.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-21T01:53:37.431 INFO:teuthology.orchestra.run.smithi016.stdout:(41/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-21T01:53:37.481 INFO:teuthology.orchestra.run.smithi016.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T01:53:37.482 INFO:teuthology.orchestra.run.smithi161.stdout:(71/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-09-21T01:53:37.507 INFO:teuthology.orchestra.run.smithi016.stdout:(43/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-21T01:53:37.573 INFO:teuthology.orchestra.run.smithi016.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-21T01:53:37.616 INFO:teuthology.orchestra.run.smithi016.stdout:(45/157): python3-cryptography-36.0.1-4.el9.x86 6.7 MB/s | 1.2 MB 00:00 2024-09-21T01:53:37.649 INFO:teuthology.orchestra.run.smithi016.stdout:(46/157): python3-pycparser-2.20-6.el9.noarch.r 951 kB/s | 135 kB 00:00 2024-09-21T01:53:37.674 INFO:teuthology.orchestra.run.smithi016.stdout:(47/157): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-09-21T01:53:37.700 INFO:teuthology.orchestra.run.smithi016.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-21T01:53:37.841 INFO:teuthology.orchestra.run.smithi161.stdout:(72/157): postgresql-server-13.16-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-09-21T01:53:37.867 INFO:teuthology.orchestra.run.smithi016.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 642 kB/s | 107 kB 00:00 2024-09-21T01:53:37.899 INFO:teuthology.orchestra.run.smithi161.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-21T01:53:37.908 INFO:teuthology.orchestra.run.smithi016.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-09-21T01:53:37.951 INFO:teuthology.orchestra.run.smithi016.stdout:(51/157): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-09-21T01:53:38.009 INFO:teuthology.orchestra.run.smithi016.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-09-21T01:53:38.034 INFO:teuthology.orchestra.run.smithi016.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-09-21T01:53:38.060 INFO:teuthology.orchestra.run.smithi016.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-09-21T01:53:38.074 INFO:teuthology.orchestra.run.smithi161.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-09-21T01:53:38.085 INFO:teuthology.orchestra.run.smithi016.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-09-21T01:53:38.110 INFO:teuthology.orchestra.run.smithi016.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-21T01:53:38.136 INFO:teuthology.orchestra.run.smithi016.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-09-21T01:53:38.137 INFO:teuthology.orchestra.run.smithi161.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 978 kB/s | 61 kB 00:00 2024-09-21T01:53:38.161 INFO:teuthology.orchestra.run.smithi016.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-21T01:53:38.186 INFO:teuthology.orchestra.run.smithi016.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-21T01:53:38.254 INFO:teuthology.orchestra.run.smithi016.stdout:(60/157): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-09-21T01:53:38.283 INFO:teuthology.orchestra.run.smithi161.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-21T01:53:38.312 INFO:teuthology.orchestra.run.smithi016.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-21T01:53:38.338 INFO:teuthology.orchestra.run.smithi016.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 293 kB/s | 24 kB 00:00 2024-09-21T01:53:38.356 INFO:teuthology.orchestra.run.smithi123.stdout:(8/157): ceph-common-19.3.0-5120.g18432218.el9. 3.8 MB/s | 20 MB 00:05 2024-09-21T01:53:38.363 INFO:teuthology.orchestra.run.smithi016.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-21T01:53:38.390 INFO:teuthology.orchestra.run.smithi123.stdout:(9/157): ceph-selinux-19.3.0-5120.g18432218.el9 764 kB/s | 25 kB 00:00 2024-09-21T01:53:38.396 INFO:teuthology.orchestra.run.smithi016.stdout:(64/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-21T01:53:38.447 INFO:teuthology.orchestra.run.smithi016.stdout:(65/157): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-21T01:53:38.459 INFO:teuthology.orchestra.run.smithi161.stdout:(77/157): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-21T01:53:38.472 INFO:teuthology.orchestra.run.smithi016.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-21T01:53:38.484 INFO:teuthology.orchestra.run.smithi161.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-09-21T01:53:38.509 INFO:teuthology.orchestra.run.smithi161.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-09-21T01:53:38.534 INFO:teuthology.orchestra.run.smithi161.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-09-21T01:53:38.568 INFO:teuthology.orchestra.run.smithi161.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-21T01:53:38.593 INFO:teuthology.orchestra.run.smithi161.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 594 kB/s | 35 kB 00:00 2024-09-21T01:53:38.623 INFO:teuthology.orchestra.run.smithi123.stdout:(10/157): ceph-radosgw-19.3.0-5120.g18432218.el 6.5 MB/s | 10 MB 00:01 2024-09-21T01:53:38.685 INFO:teuthology.orchestra.run.smithi161.stdout:(83/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.4 MB/s | 4.9 MB 00:02 2024-09-21T01:53:38.690 INFO:teuthology.orchestra.run.smithi123.stdout:(11/157): cephfs-mirror-19.3.0-5120.g18432218.e 3.3 MB/s | 226 kB 00:00 2024-09-21T01:53:38.724 INFO:teuthology.orchestra.run.smithi123.stdout:(12/157): cephfs-shell-19.3.0-5120.g18432218.el 817 kB/s | 27 kB 00:00 2024-09-21T01:53:38.757 INFO:teuthology.orchestra.run.smithi123.stdout:(13/157): libcephfs-devel-19.3.0-5120.g18432218 951 kB/s | 32 kB 00:00 2024-09-21T01:53:38.769 INFO:teuthology.orchestra.run.smithi161.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-09-21T01:53:38.861 INFO:teuthology.orchestra.run.smithi161.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-09-21T01:53:38.886 INFO:teuthology.orchestra.run.smithi161.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 659 kB/s | 77 kB 00:00 2024-09-21T01:53:38.891 INFO:teuthology.orchestra.run.smithi123.stdout:(14/157): libcephfs2-19.3.0-5120.g18432218.el9. 5.6 MB/s | 766 kB 00:00 2024-09-21T01:53:38.911 INFO:teuthology.orchestra.run.smithi161.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 838 kB/s | 42 kB 00:00 2024-09-21T01:53:38.950 INFO:teuthology.orchestra.run.smithi123.stdout:(15/157): libcephsqlite-19.3.0-5120.g18432218.e 2.6 MB/s | 158 kB 00:00 2024-09-21T01:53:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:(16/157): librados-devel-19.3.0-5120.g18432218. 2.1 MB/s | 127 kB 00:00 2024-09-21T01:53:39.045 INFO:teuthology.orchestra.run.smithi161.stdout:(88/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-21T01:53:39.070 INFO:teuthology.orchestra.run.smithi161.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-21T01:53:39.104 INFO:teuthology.orchestra.run.smithi161.stdout:(90/157): python3-protobuf-3.14.0-13.el9.noarch 1.2 MB/s | 269 kB 00:00 2024-09-21T01:53:39.109 INFO:teuthology.orchestra.run.smithi123.stdout:(17/157): libradosstriper1-19.3.0-5120.g1843221 5.0 MB/s | 507 kB 00:00 2024-09-21T01:53:39.129 INFO:teuthology.orchestra.run.smithi161.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-21T01:53:39.154 INFO:teuthology.orchestra.run.smithi161.stdout:(92/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-09-21T01:53:39.179 INFO:teuthology.orchestra.run.smithi161.stdout:(93/157): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-09-21T01:53:39.180 INFO:teuthology.orchestra.run.smithi016.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-09-21T01:53:39.205 INFO:teuthology.orchestra.run.smithi161.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-09-21T01:53:39.230 INFO:teuthology.orchestra.run.smithi161.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 877 kB/s | 44 kB 00:00 2024-09-21T01:53:39.263 INFO:teuthology.orchestra.run.smithi161.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-21T01:53:39.280 INFO:teuthology.orchestra.run.smithi161.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-09-21T01:53:39.314 INFO:teuthology.orchestra.run.smithi161.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-21T01:53:39.364 INFO:teuthology.orchestra.run.smithi161.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T01:53:39.415 INFO:teuthology.orchestra.run.smithi161.stdout:(100/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-21T01:53:39.465 INFO:teuthology.orchestra.run.smithi161.stdout:(101/157): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-09-21T01:53:39.482 INFO:teuthology.orchestra.run.smithi161.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 949 kB/s | 64 kB 00:00 2024-09-21T01:53:39.507 INFO:teuthology.orchestra.run.smithi161.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-09-21T01:53:39.734 INFO:teuthology.orchestra.run.smithi123.stdout:(18/157): ceph-osd-19.3.0-5120.g18432218.el9.x8 5.5 MB/s | 18 MB 00:03 2024-09-21T01:53:39.784 INFO:teuthology.orchestra.run.smithi123.stdout:(19/157): python3-ceph-argparse-19.3.0-5120.g18 911 kB/s | 45 kB 00:00 2024-09-21T01:53:39.859 INFO:teuthology.orchestra.run.smithi123.stdout:(20/157): librgw2-19.3.0-5120.g18432218.el9.x86 7.0 MB/s | 5.3 MB 00:00 2024-09-21T01:53:39.885 INFO:teuthology.orchestra.run.smithi123.stdout:(21/157): python3-ceph-common-19.3.0-5120.g1843 1.4 MB/s | 148 kB 00:00 2024-09-21T01:53:39.910 INFO:teuthology.orchestra.run.smithi123.stdout:(22/157): python3-cephfs-19.3.0-5120.g18432218. 3.2 MB/s | 161 kB 00:00 2024-09-21T01:53:39.952 INFO:teuthology.orchestra.run.smithi123.stdout:(23/157): python3-rados-19.3.0-5120.g18432218.e 4.7 MB/s | 321 kB 00:00 2024-09-21T01:53:39.984 INFO:teuthology.orchestra.run.smithi016.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-09-21T01:53:40.002 INFO:teuthology.orchestra.run.smithi123.stdout:(24/157): python3-rbd-19.3.0-5120.g18432218.el9 3.3 MB/s | 307 kB 00:00 2024-09-21T01:53:40.008 INFO:teuthology.orchestra.run.smithi161.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-09-21T01:53:40.028 INFO:teuthology.orchestra.run.smithi123.stdout:(25/157): python3-rgw-19.3.0-5120.g18432218.el9 1.3 MB/s | 99 kB 00:00 2024-09-21T01:53:40.053 INFO:teuthology.orchestra.run.smithi123.stdout:(26/157): rbd-fuse-19.3.0-5120.g18432218.el9.x8 1.7 MB/s | 86 kB 00:00 2024-09-21T01:53:40.058 INFO:teuthology.orchestra.run.smithi161.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T01:53:40.091 INFO:teuthology.orchestra.run.smithi161.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-21T01:53:40.103 INFO:teuthology.orchestra.run.smithi123.stdout:(27/157): rbd-nbd-19.3.0-5120.g18432218.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-21T01:53:40.108 INFO:teuthology.orchestra.run.smithi161.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-21T01:53:40.145 INFO:teuthology.orchestra.run.smithi123.stdout:(28/157): ceph-grafana-dashboards-19.3.0-5120.g 827 kB/s | 34 kB 00:00 2024-09-21T01:53:40.195 INFO:teuthology.orchestra.run.smithi123.stdout:(29/157): ceph-mgr-cephadm-19.3.0-5120.g1843221 3.0 MB/s | 154 kB 00:00 2024-09-21T01:53:40.250 INFO:teuthology.orchestra.run.smithi161.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-09-21T01:53:40.267 INFO:teuthology.orchestra.run.smithi161.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-21T01:53:40.284 INFO:teuthology.orchestra.run.smithi161.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-21T01:53:40.309 INFO:teuthology.orchestra.run.smithi161.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T01:53:40.334 INFO:teuthology.orchestra.run.smithi161.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-09-21T01:53:40.394 INFO:teuthology.orchestra.run.smithi161.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-21T01:53:40.464 INFO:teuthology.orchestra.run.smithi016.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 1.9 MB/s | 910 kB 00:00 2024-09-21T01:53:40.464 INFO:teuthology.orchestra.run.smithi123.stdout:(30/157): rbd-mirror-19.3.0-5120.g18432218.el9. 7.9 MB/s | 3.1 MB 00:00 2024-09-21T01:53:40.464 INFO:teuthology.orchestra.run.smithi161.stdout:(114/157): protobuf-compiler-3.14.0-13.el9.x86_ 939 kB/s | 863 kB 00:00 2024-09-21T01:53:40.464 INFO:teuthology.orchestra.run.smithi161.stdout:(115/157): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-09-21T01:53:40.486 INFO:teuthology.orchestra.run.smithi161.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.4 MB/s | 32 kB 00:00 2024-09-21T01:53:40.513 INFO:teuthology.orchestra.run.smithi161.stdout:(117/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 519 kB/s | 43 kB 00:00 2024-09-21T01:53:40.536 INFO:teuthology.orchestra.run.smithi161.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-21T01:53:40.564 INFO:teuthology.orchestra.run.smithi161.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-09-21T01:53:40.587 INFO:teuthology.orchestra.run.smithi161.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-21T01:53:40.606 INFO:teuthology.orchestra.run.smithi016.stdout:(70/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.3 MB/s | 4.9 MB 00:02 2024-09-21T01:53:40.612 INFO:teuthology.orchestra.run.smithi161.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-09-21T01:53:40.631 INFO:teuthology.orchestra.run.smithi016.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 749 kB/s | 137 kB 00:00 2024-09-21T01:53:40.637 INFO:teuthology.orchestra.run.smithi161.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-21T01:53:40.671 INFO:teuthology.orchestra.run.smithi161.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-21T01:53:40.688 INFO:teuthology.orchestra.run.smithi161.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 638 kB/s | 11 kB 00:00 2024-09-21T01:53:40.705 INFO:teuthology.orchestra.run.smithi161.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-09-21T01:53:40.730 INFO:teuthology.orchestra.run.smithi161.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 925 kB/s | 23 kB 00:00 2024-09-21T01:53:40.747 INFO:teuthology.orchestra.run.smithi161.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 1.1 MB/s | 19 kB 00:00 2024-09-21T01:53:40.766 INFO:teuthology.orchestra.run.smithi016.stdout:(72/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 1.9 MB/s | 4.4 MB 00:02 2024-09-21T01:53:40.798 INFO:teuthology.orchestra.run.smithi161.stdout:(128/157): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-09-21T01:53:40.823 INFO:teuthology.orchestra.run.smithi161.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-09-21T01:53:40.848 INFO:teuthology.orchestra.run.smithi161.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T01:53:40.849 INFO:teuthology.orchestra.run.smithi016.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-09-21T01:53:40.873 INFO:teuthology.orchestra.run.smithi161.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T01:53:40.907 INFO:teuthology.orchestra.run.smithi161.stdout:(132/157): python3-kubernetes-26.1.0-2.el9.noar 9.3 MB/s | 1.0 MB 00:00 2024-09-21T01:53:40.933 INFO:teuthology.orchestra.run.smithi161.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 984 kB/s | 58 kB 00:00 2024-09-21T01:53:40.958 INFO:teuthology.orchestra.run.smithi161.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 660 kB/s | 16 kB 00:00 2024-09-21T01:53:40.983 INFO:teuthology.orchestra.run.smithi161.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-21T01:53:41.008 INFO:teuthology.orchestra.run.smithi161.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-21T01:53:41.033 INFO:teuthology.orchestra.run.smithi161.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-09-21T01:53:41.059 INFO:teuthology.orchestra.run.smithi161.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-21T01:53:41.084 INFO:teuthology.orchestra.run.smithi161.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T01:53:41.109 INFO:teuthology.orchestra.run.smithi161.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-21T01:53:41.135 INFO:teuthology.orchestra.run.smithi161.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-09-21T01:53:41.160 INFO:teuthology.orchestra.run.smithi161.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-21T01:53:41.185 INFO:teuthology.orchestra.run.smithi161.stdout:(143/157): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-21T01:53:41.199 INFO:teuthology.orchestra.run.smithi016.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-21T01:53:41.211 INFO:teuthology.orchestra.run.smithi161.stdout:(144/157): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T01:53:41.236 INFO:teuthology.orchestra.run.smithi161.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 807 kB/s | 20 kB 00:00 2024-09-21T01:53:41.238 INFO:teuthology.orchestra.run.smithi123.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 9.0 MB/s | 7.4 MB 00:00 2024-09-21T01:53:41.249 INFO:teuthology.orchestra.run.smithi016.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-21T01:53:41.261 INFO:teuthology.orchestra.run.smithi161.stdout:(146/157): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-09-21T01:53:41.286 INFO:teuthology.orchestra.run.smithi161.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T01:53:41.305 INFO:teuthology.orchestra.run.smithi123.stdout:(32/157): ceph-mgr-modules-core-19.3.0-5120.g18 4.3 MB/s | 292 kB 00:00 2024-09-21T01:53:41.346 INFO:teuthology.orchestra.run.smithi123.stdout:(33/157): ceph-mgr-rook-19.3.0-5120.g18432218.e 1.2 MB/s | 49 kB 00:00 2024-09-21T01:53:41.353 INFO:teuthology.orchestra.run.smithi161.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-21T01:53:41.380 INFO:teuthology.orchestra.run.smithi123.stdout:(34/157): ceph-prometheus-alerts-19.3.0-5120.g1 494 kB/s | 16 kB 00:00 2024-09-21T01:53:41.391 INFO:teuthology.orchestra.run.smithi016.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-21T01:53:41.447 INFO:teuthology.orchestra.run.smithi123.stdout:(35/157): ceph-volume-19.3.0-5120.g18432218.el9 4.2 MB/s | 285 kB 00:00 2024-09-21T01:53:41.533 INFO:teuthology.orchestra.run.smithi016.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-09-21T01:53:41.575 INFO:teuthology.orchestra.run.smithi016.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 615 kB/s | 26 kB 00:00 2024-09-21T01:53:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:(36/157): cephadm-19.3.0-5120.g18432218.el9.noa 5.1 MB/s | 784 kB 00:00 2024-09-21T01:53:41.617 INFO:teuthology.orchestra.run.smithi016.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 462 kB/s | 19 kB 00:00 2024-09-21T01:53:41.620 INFO:teuthology.orchestra.run.smithi161.stdout:(149/157): librados2-19.3.0-5120.g18432218.el9. 10 MB/s | 3.4 MB 00:00 2024-09-21T01:53:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:(37/157): cephfs-top-19.3.0-5120.g18432218.el9. 445 kB/s | 18 kB 00:00 2024-09-21T01:53:41.679 INFO:teuthology.orchestra.run.smithi161.stdout:(150/157): librbd1-19.3.0-5120.g18432218.el9.x8 9.7 MB/s | 3.2 MB 00:00 2024-09-21T01:53:41.734 INFO:teuthology.orchestra.run.smithi016.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-21T01:53:41.776 INFO:teuthology.orchestra.run.smithi016.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 837 kB/s | 35 kB 00:00 2024-09-21T01:53:41.781 INFO:teuthology.orchestra.run.smithi123.stdout:(38/157): fuse-2.9.9-16.el9.x86_64.rpm 563 kB/s | 80 kB 00:00 2024-09-21T01:53:41.904 INFO:teuthology.orchestra.run.smithi161.stdout:(151/157): NetworkManager-1.51.0-1.el9.x86_64.r 8.2 MB/s | 2.3 MB 00:00 2024-09-21T01:53:41.946 INFO:teuthology.orchestra.run.smithi161.stdout:(152/157): NetworkManager-libnm-1.51.0-1.el9.x8 6.8 MB/s | 1.8 MB 00:00 2024-09-21T01:53:41.971 INFO:teuthology.orchestra.run.smithi161.stdout:(153/157): NetworkManager-team-1.51.0-1.el9.x86 612 kB/s | 41 kB 00:00 2024-09-21T01:53:41.990 INFO:teuthology.orchestra.run.smithi123.stdout:(39/157): ceph-mgr-dashboard-19.3.0-5120.g18432 7.9 MB/s | 14 MB 00:01 2024-09-21T01:53:41.997 INFO:teuthology.orchestra.run.smithi161.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 4.8 MB/s | 249 kB 00:00 2024-09-21T01:53:42.039 INFO:teuthology.orchestra.run.smithi161.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 20 MB/s | 1.4 MB 00:00 2024-09-21T01:53:42.089 INFO:teuthology.orchestra.run.smithi161.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-09-21T01:53:42.107 INFO:teuthology.orchestra.run.smithi123.stdout:(40/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-21T01:53:42.190 INFO:teuthology.orchestra.run.smithi123.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 956 kB/s | 191 kB 00:00 2024-09-21T01:53:42.215 INFO:teuthology.orchestra.run.smithi123.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-09-21T01:53:42.266 INFO:teuthology.orchestra.run.smithi123.stdout:(43/157): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T01:53:42.358 INFO:teuthology.orchestra.run.smithi123.stdout:(44/157): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-09-21T01:53:42.542 INFO:teuthology.orchestra.run.smithi123.stdout:(45/157): python3-cryptography-36.0.1-4.el9.x86 3.5 MB/s | 1.2 MB 00:00 2024-09-21T01:53:42.567 INFO:teuthology.orchestra.run.smithi123.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 599 kB/s | 125 kB 00:00 2024-09-21T01:53:42.625 INFO:teuthology.orchestra.run.smithi123.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-21T01:53:43.018 INFO:teuthology.orchestra.run.smithi123.stdout:(48/157): ceph-test-19.3.0-5120.g18432218.el9.x 8.6 MB/s | 40 MB 00:04 2024-09-21T01:53:43.051 INFO:teuthology.orchestra.run.smithi123.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 1.4 MB/s | 689 kB 00:00 2024-09-21T01:53:43.085 INFO:teuthology.orchestra.run.smithi123.stdout:(50/157): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-09-21T01:53:43.093 INFO:teuthology.orchestra.run.smithi016.stdout:(82/157): postgresql-server-13.16-1.el9.x86_64. 2.3 MB/s | 5.8 MB 00:02 2024-09-21T01:53:43.110 INFO:teuthology.orchestra.run.smithi123.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-09-21T01:53:43.160 INFO:teuthology.orchestra.run.smithi123.stdout:(52/157): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-09-21T01:53:43.185 INFO:teuthology.orchestra.run.smithi123.stdout:(53/157): boost-program-options-1.75.0-8.el9.x8 639 kB/s | 107 kB 00:00 2024-09-21T01:53:43.211 INFO:teuthology.orchestra.run.smithi123.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-09-21T01:53:43.236 INFO:teuthology.orchestra.run.smithi123.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-09-21T01:53:43.243 INFO:teuthology.orchestra.run.smithi016.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-09-21T01:53:43.261 INFO:teuthology.orchestra.run.smithi123.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-21T01:53:43.320 INFO:teuthology.orchestra.run.smithi123.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.0 MB/s | 171 kB 00:00 2024-09-21T01:53:43.345 INFO:teuthology.orchestra.run.smithi123.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-21T01:53:43.370 INFO:teuthology.orchestra.run.smithi123.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-09-21T01:53:43.385 INFO:teuthology.orchestra.run.smithi016.stdout:(84/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-09-21T01:53:43.469 INFO:teuthology.orchestra.run.smithi016.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 924 kB/s | 77 kB 00:00 2024-09-21T01:53:43.519 INFO:teuthology.orchestra.run.smithi016.stdout:(86/157): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-09-21T01:53:43.521 INFO:teuthology.orchestra.run.smithi123.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-21T01:53:43.546 INFO:teuthology.orchestra.run.smithi123.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 3.2 MB/s | 581 kB 00:00 2024-09-21T01:53:43.571 INFO:teuthology.orchestra.run.smithi123.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 488 kB/s | 24 kB 00:00 2024-09-21T01:53:43.596 INFO:teuthology.orchestra.run.smithi123.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-09-21T01:53:43.680 INFO:teuthology.orchestra.run.smithi123.stdout:(64/157): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-09-21T01:53:43.686 INFO:teuthology.orchestra.run.smithi016.stdout:(87/157): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-09-21T01:53:43.713 INFO:teuthology.orchestra.run.smithi123.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-21T01:53:43.731 INFO:teuthology.orchestra.run.smithi123.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 767 kB/s | 38 kB 00:00 2024-09-21T01:53:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T01:53:43.979 INFO:teuthology.orchestra.run.smithi016.stdout:(89/157): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-09-21T01:53:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 279 kB 00:00 2024-09-21T01:53:44.063 INFO:teuthology.orchestra.run.smithi016.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 611 kB/s | 51 kB 00:00 2024-09-21T01:53:44.088 INFO:teuthology.orchestra.run.smithi016.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-21T01:53:44.114 INFO:teuthology.orchestra.run.smithi123.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.1 MB/s | 3.0 MB 00:00 2024-09-21T01:53:44.138 INFO:teuthology.orchestra.run.smithi016.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-09-21T01:53:44.197 INFO:teuthology.orchestra.run.smithi016.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 759 kB/s | 44 kB 00:00 2024-09-21T01:53:44.353 INFO:teuthology.orchestra.run.smithi016.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-21T01:53:44.448 INFO:teuthology.orchestra.run.smithi016.stdout:(96/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-09-21T01:53:44.473 INFO:teuthology.orchestra.run.smithi016.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 286 kB/s | 36 kB 00:00 2024-09-21T01:53:44.489 INFO:teuthology.orchestra.run.smithi123.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-09-21T01:53:44.498 INFO:teuthology.orchestra.run.smithi016.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-09-21T01:53:44.548 INFO:teuthology.orchestra.run.smithi016.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T01:53:44.573 INFO:teuthology.orchestra.run.smithi123.stdout:(69/157): postgresql-13.16-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-09-21T01:53:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:(100/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 959 kB/s | 96 kB 00:00 2024-09-21T01:53:44.656 INFO:teuthology.orchestra.run.smithi123.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 5.3 MB/s | 910 kB 00:00 2024-09-21T01:53:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:(101/157): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T01:53:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:(102/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-09-21T01:53:44.749 INFO:teuthology.orchestra.run.smithi016.stdout:(103/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 633 kB/s | 64 kB 00:00 2024-09-21T01:53:44.765 INFO:teuthology.orchestra.run.smithi123.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 715 kB/s | 137 kB 00:00 2024-09-21T01:53:44.824 INFO:teuthology.orchestra.run.smithi016.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-09-21T01:53:44.875 INFO:teuthology.orchestra.run.smithi016.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T01:53:44.890 INFO:teuthology.orchestra.run.smithi123.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 4.2 MB/s | 4.9 MB 00:01 2024-09-21T01:53:44.909 INFO:teuthology.orchestra.run.smithi016.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-21T01:53:44.925 INFO:teuthology.orchestra.run.smithi016.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-21T01:53:44.941 INFO:teuthology.orchestra.run.smithi123.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-21T01:53:45.043 INFO:teuthology.orchestra.run.smithi016.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-09-21T01:53:45.059 INFO:teuthology.orchestra.run.smithi016.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.6 MB/s | 26 kB 00:00 2024-09-21T01:53:45.085 INFO:teuthology.orchestra.run.smithi016.stdout:(110/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-09-21T01:53:45.110 INFO:teuthology.orchestra.run.smithi016.stdout:(111/157): liboath-2.6.7-2.el9.x86_64.rpm 963 kB/s | 49 kB 00:00 2024-09-21T01:53:45.136 INFO:teuthology.orchestra.run.smithi016.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 6.4 MB/s | 151 kB 00:00 2024-09-21T01:53:45.161 INFO:teuthology.orchestra.run.smithi016.stdout:(113/157): libunwind-1.6.2-1.el9.x86_64.rpm 892 kB/s | 67 kB 00:00 2024-09-21T01:53:45.175 INFO:teuthology.orchestra.run.smithi123.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-09-21T01:53:45.194 INFO:teuthology.orchestra.run.smithi016.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-21T01:53:45.225 INFO:teuthology.orchestra.run.smithi123.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-21T01:53:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-21T01:53:45.245 INFO:teuthology.orchestra.run.smithi016.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.9 MB/s | 32 kB 00:00 2024-09-21T01:53:45.262 INFO:teuthology.orchestra.run.smithi016.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 856 kB/s | 14 kB 00:00 2024-09-21T01:53:45.296 INFO:teuthology.orchestra.run.smithi016.stdout:(118/157): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-09-21T01:53:45.321 INFO:teuthology.orchestra.run.smithi016.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-21T01:53:45.346 INFO:teuthology.orchestra.run.smithi016.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-21T01:53:45.358 INFO:teuthology.orchestra.run.smithi123.stdout:(76/157): postgresql-server-13.16-1.el9.x86_64. 8.2 MB/s | 5.8 MB 00:00 2024-09-21T01:53:45.372 INFO:teuthology.orchestra.run.smithi016.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-09-21T01:53:45.384 INFO:teuthology.orchestra.run.smithi123.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-09-21T01:53:45.397 INFO:teuthology.orchestra.run.smithi016.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-21T01:53:45.417 INFO:teuthology.orchestra.run.smithi123.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-09-21T01:53:45.431 INFO:teuthology.orchestra.run.smithi016.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-21T01:53:45.442 INFO:teuthology.orchestra.run.smithi123.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 437 kB/s | 26 kB 00:00 2024-09-21T01:53:45.456 INFO:teuthology.orchestra.run.smithi016.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-21T01:53:45.476 INFO:teuthology.orchestra.run.smithi123.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-09-21T01:53:45.526 INFO:teuthology.orchestra.run.smithi123.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-21T01:53:45.552 INFO:teuthology.orchestra.run.smithi123.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 461 kB/s | 35 kB 00:00 2024-09-21T01:53:45.598 INFO:teuthology.orchestra.run.smithi016.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 125 kB/s | 18 kB 00:00 2024-09-21T01:53:45.610 INFO:teuthology.orchestra.run.smithi123.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-21T01:53:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-09-21T01:53:45.649 INFO:teuthology.orchestra.run.smithi016.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 778 kB/s | 19 kB 00:00 2024-09-21T01:53:45.674 INFO:teuthology.orchestra.run.smithi016.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 789 kB/s | 20 kB 00:00 2024-09-21T01:53:45.677 INFO:teuthology.orchestra.run.smithi123.stdout:(84/157): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-09-21T01:53:45.728 INFO:teuthology.orchestra.run.smithi123.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-21T01:53:45.732 INFO:teuthology.orchestra.run.smithi016.stdout:(129/157): python3-grpcio-1.46.7-10.el9.x86_64. 5.7 MB/s | 2.0 MB 00:00 2024-09-21T01:53:45.758 INFO:teuthology.orchestra.run.smithi016.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T01:53:45.778 INFO:teuthology.orchestra.run.smithi123.stdout:(86/157): python3-prettytable-0.7.2-27.el9.noar 851 kB/s | 42 kB 00:00 2024-09-21T01:53:45.783 INFO:teuthology.orchestra.run.smithi016.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T01:53:45.809 INFO:teuthology.orchestra.run.smithi016.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-21T01:53:45.836 INFO:teuthology.orchestra.run.smithi123.stdout:(87/157): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-21T01:53:45.842 INFO:teuthology.orchestra.run.smithi016.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-21T01:53:45.867 INFO:teuthology.orchestra.run.smithi016.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 662 kB/s | 16 kB 00:00 2024-09-21T01:53:45.895 INFO:teuthology.orchestra.run.smithi123.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-21T01:53:45.909 INFO:teuthology.orchestra.run.smithi016.stdout:(135/157): python3-kubernetes-26.1.0-2.el9.noar 4.3 MB/s | 1.0 MB 00:00 2024-09-21T01:53:45.935 INFO:teuthology.orchestra.run.smithi016.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-09-21T01:53:45.951 INFO:teuthology.orchestra.run.smithi016.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 675 kB/s | 28 kB 00:00 2024-09-21T01:53:45.954 INFO:teuthology.orchestra.run.smithi123.stdout:(89/157): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-21T01:53:45.968 INFO:teuthology.orchestra.run.smithi016.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 920 kB/s | 31 kB 00:00 2024-09-21T01:53:45.985 INFO:teuthology.orchestra.run.smithi016.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-21T01:53:46.004 INFO:teuthology.orchestra.run.smithi123.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T01:53:46.011 INFO:teuthology.orchestra.run.smithi016.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-09-21T01:53:46.036 INFO:teuthology.orchestra.run.smithi016.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-09-21T01:53:46.054 INFO:teuthology.orchestra.run.smithi123.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-21T01:53:46.062 INFO:teuthology.orchestra.run.smithi016.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-21T01:53:46.087 INFO:teuthology.orchestra.run.smithi016.stdout:(143/157): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-21T01:53:46.116 INFO:teuthology.orchestra.run.smithi016.stdout:(144/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.9 MB/s | 230 kB 00:00 2024-09-21T01:53:46.141 INFO:teuthology.orchestra.run.smithi016.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 802 kB/s | 20 kB 00:00 2024-09-21T01:53:46.166 INFO:teuthology.orchestra.run.smithi016.stdout:(146/157): python3-werkzeug-2.0.3-3.el9.1.noarc 5.3 MB/s | 427 kB 00:00 2024-09-21T01:53:46.192 INFO:teuthology.orchestra.run.smithi016.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T01:53:46.242 INFO:teuthology.orchestra.run.smithi016.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-09-21T01:53:46.484 INFO:teuthology.orchestra.run.smithi016.stdout:(149/157): librados2-19.3.0-5120.g18432218.el9. 12 MB/s | 3.4 MB 00:00 2024-09-21T01:53:46.530 INFO:teuthology.orchestra.run.smithi123.stdout:(92/157): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-09-21T01:53:46.567 INFO:teuthology.orchestra.run.smithi016.stdout:(150/157): librbd1-19.3.0-5120.g18432218.el9.x8 9.7 MB/s | 3.2 MB 00:00 2024-09-21T01:53:46.588 INFO:teuthology.orchestra.run.smithi123.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 718 kB/s | 42 kB 00:00 2024-09-21T01:53:46.638 INFO:teuthology.orchestra.run.smithi123.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-09-21T01:53:46.718 INFO:teuthology.orchestra.run.smithi016.stdout:(151/157): NetworkManager-libnm-1.51.0-1.el9.x8 12 MB/s | 1.8 MB 00:00 2024-09-21T01:53:46.730 INFO:teuthology.orchestra.run.smithi123.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-21T01:53:46.801 INFO:teuthology.orchestra.run.smithi016.stdout:(152/157): NetworkManager-1.51.0-1.el9.x86_64.r 7.3 MB/s | 2.3 MB 00:00 2024-09-21T01:53:46.827 INFO:teuthology.orchestra.run.smithi016.stdout:(153/157): NetworkManager-team-1.51.0-1.el9.x86 377 kB/s | 41 kB 00:00 2024-09-21T01:53:46.830 INFO:teuthology.orchestra.run.smithi123.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 359 kB/s | 36 kB 00:00 2024-09-21T01:53:46.881 INFO:teuthology.orchestra.run.smithi123.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-09-21T01:53:46.902 INFO:teuthology.orchestra.run.smithi016.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 2.4 MB/s | 249 kB 00:00 2024-09-21T01:53:46.952 INFO:teuthology.orchestra.run.smithi016.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 11 MB/s | 1.4 MB 00:00 2024-09-21T01:53:46.998 INFO:teuthology.orchestra.run.smithi123.stdout:(98/157): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-21T01:53:47.086 INFO:teuthology.orchestra.run.smithi016.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-09-21T01:53:47.090 INFO:teuthology.orchestra.run.smithi123.stdout:(99/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.3 MB/s | 5.1 MB 00:01 2024-09-21T01:53:47.107 INFO:teuthology.orchestra.run.smithi123.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 533 kB/s | 58 kB 00:00 2024-09-21T01:53:47.140 INFO:teuthology.orchestra.run.smithi123.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-21T01:53:47.157 INFO:teuthology.orchestra.run.smithi123.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T01:53:47.191 INFO:teuthology.orchestra.run.smithi123.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 890 kB/s | 45 kB 00:00 2024-09-21T01:53:47.647 INFO:teuthology.orchestra.run.smithi161.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.9 MB/s | 16 MB 00:08 2024-09-21T01:53:47.653 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:53:47.653 INFO:teuthology.orchestra.run.smithi161.stdout:Total 13 MB/s | 228 MB 00:17 2024-09-21T01:53:47.733 INFO:teuthology.orchestra.run.smithi123.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-09-21T01:53:47.783 INFO:teuthology.orchestra.run.smithi123.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-21T01:53:47.808 INFO:teuthology.orchestra.run.smithi123.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 12 MB/s | 308 kB 00:00 2024-09-21T01:53:47.825 INFO:teuthology.orchestra.run.smithi123.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-21T01:53:47.934 INFO:teuthology.orchestra.run.smithi123.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T01:53:47.951 INFO:teuthology.orchestra.run.smithi123.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-21T01:53:47.968 INFO:teuthology.orchestra.run.smithi123.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-09-21T01:53:47.993 INFO:teuthology.orchestra.run.smithi123.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T01:53:48.018 INFO:teuthology.orchestra.run.smithi123.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-21T01:53:48.060 INFO:teuthology.orchestra.run.smithi123.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-21T01:53:48.236 INFO:teuthology.orchestra.run.smithi123.stdout:(114/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.5 MB/s | 16 MB 00:02 2024-09-21T01:53:48.277 INFO:teuthology.orchestra.run.smithi123.stdout:(115/157): protobuf-compiler-3.14.0-13.el9.x86_ 795 kB/s | 863 kB 00:01 2024-09-21T01:53:48.303 INFO:teuthology.orchestra.run.smithi123.stdout:(116/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 653 kB/s | 43 kB 00:00 2024-09-21T01:53:48.328 INFO:teuthology.orchestra.run.smithi123.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 607 kB/s | 14 kB 00:00 2024-09-21T01:53:48.345 INFO:teuthology.orchestra.run.smithi123.stdout:(118/157): python3-cachetools-4.2.4-1.el9.noarc 491 kB/s | 32 kB 00:00 2024-09-21T01:53:48.370 INFO:teuthology.orchestra.run.smithi123.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-21T01:53:48.404 INFO:teuthology.orchestra.run.smithi123.stdout:(120/157): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-09-21T01:53:48.429 INFO:teuthology.orchestra.run.smithi123.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-09-21T01:53:48.454 INFO:teuthology.orchestra.run.smithi123.stdout:(122/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-09-21T01:53:48.479 INFO:teuthology.orchestra.run.smithi123.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 2.9 MB/s | 220 kB 00:00 2024-09-21T01:53:48.505 INFO:teuthology.orchestra.run.smithi123.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-09-21T01:53:48.530 INFO:teuthology.orchestra.run.smithi123.stdout:(125/157): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-21T01:53:48.555 INFO:teuthology.orchestra.run.smithi123.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-09-21T01:53:48.581 INFO:teuthology.orchestra.run.smithi123.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-09-21T01:53:48.606 INFO:teuthology.orchestra.run.smithi123.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-21T01:53:48.631 INFO:teuthology.orchestra.run.smithi123.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-21T01:53:48.656 INFO:teuthology.orchestra.run.smithi123.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-21T01:53:48.682 INFO:teuthology.orchestra.run.smithi123.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-21T01:53:48.707 INFO:teuthology.orchestra.run.smithi123.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-09-21T01:53:48.741 INFO:teuthology.orchestra.run.smithi123.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 7.9 MB/s | 272 kB 00:00 2024-09-21T01:53:48.774 INFO:teuthology.orchestra.run.smithi123.stdout:(134/157): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-09-21T01:53:48.800 INFO:teuthology.orchestra.run.smithi123.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-09-21T01:53:48.825 INFO:teuthology.orchestra.run.smithi123.stdout:(136/157): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-21T01:53:48.850 INFO:teuthology.orchestra.run.smithi123.stdout:(137/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-09-21T01:53:48.856 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-09-21T01:53:48.875 INFO:teuthology.orchestra.run.smithi123.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-21T01:53:48.901 INFO:teuthology.orchestra.run.smithi123.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T01:53:48.926 INFO:teuthology.orchestra.run.smithi123.stdout:(140/157): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-21T01:53:48.943 INFO:teuthology.orchestra.run.smithi123.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 854 kB/s | 36 kB 00:00 2024-09-21T01:53:48.968 INFO:teuthology.orchestra.run.smithi123.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-21T01:53:49.012 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-09-21T01:53:49.013 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-09-21T01:53:49.035 INFO:teuthology.orchestra.run.smithi123.stdout:(143/157): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-09-21T01:53:49.060 INFO:teuthology.orchestra.run.smithi123.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 980 kB/s | 90 kB 00:00 2024-09-21T01:53:49.110 INFO:teuthology.orchestra.run.smithi123.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-09-21T01:53:49.127 INFO:teuthology.orchestra.run.smithi123.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-09-21T01:53:49.161 INFO:teuthology.orchestra.run.smithi123.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-21T01:53:49.194 INFO:teuthology.orchestra.run.smithi123.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-09-21T01:53:49.245 INFO:teuthology.orchestra.run.smithi123.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-21T01:53:49.587 INFO:teuthology.orchestra.run.smithi123.stdout:(150/157): librados2-19.3.0-5120.g18432218.el9. 8.0 MB/s | 3.4 MB 00:00 2024-09-21T01:53:49.653 INFO:teuthology.orchestra.run.smithi123.stdout:(151/157): librbd1-19.3.0-5120.g18432218.el9.x8 6.9 MB/s | 3.2 MB 00:00 2024-09-21T01:53:49.695 INFO:teuthology.orchestra.run.smithi123.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 984 kB/s | 41 kB 00:00 2024-09-21T01:53:49.804 INFO:teuthology.orchestra.run.smithi123.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 2.2 MB/s | 249 kB 00:00 2024-09-21T01:53:49.904 INFO:teuthology.orchestra.run.smithi123.stdout:(154/157): NetworkManager-1.51.0-1.el9.x86_64.r 3.5 MB/s | 2.3 MB 00:00 2024-09-21T01:53:50.146 INFO:teuthology.orchestra.run.smithi123.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 4.0 MB/s | 1.4 MB 00:00 2024-09-21T01:53:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:(156/157): NetworkManager-libnm-1.51.0-1.el9.x8 3.0 MB/s | 1.8 MB 00:00 2024-09-21T01:53:50.430 INFO:teuthology.orchestra.run.smithi123.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 4.6 MB/s | 2.4 MB 00:00 2024-09-21T01:53:50.435 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:53:50.435 INFO:teuthology.orchestra.run.smithi123.stdout:Total 13 MB/s | 228 MB 00:17 2024-09-21T01:53:51.245 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-09-21T01:53:51.245 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-09-21T01:53:51.670 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-09-21T01:53:51.824 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-09-21T01:53:51.824 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-09-21T01:53:52.561 INFO:teuthology.orchestra.run.smithi016.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.9 MB/s | 16 MB 00:08 2024-09-21T01:53:52.566 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:53:52.567 INFO:teuthology.orchestra.run.smithi016.stdout:Total 11 MB/s | 228 MB 00:20 2024-09-21T01:53:53.826 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-21T01:53:53.999 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-21T01:53:53.999 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-21T01:53:54.065 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-09-21T01:53:54.066 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-09-21T01:53:56.370 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-21T01:53:56.370 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-21T01:53:57.774 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-09-21T01:53:57.867 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T01:53:57.906 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T01:53:57.959 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T01:53:57.993 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T01:53:58.018 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T01:53:58.067 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T01:53:58.403 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T01:53:58.536 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:53:58.645 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:53:58.678 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:53:58.753 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:53:58.783 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:53:58.812 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T01:53:58.990 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T01:53:59.047 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T01:53:59.075 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:53:59.181 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:53:59.209 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:53:59.255 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:53:59.277 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:53:59.299 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:53:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:53:59.639 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:53:59.758 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T01:53:59.853 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:53:59.943 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:00.281 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T01:54:00.322 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T01:54:00.357 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T01:54:00.437 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T01:54:00.491 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T01:54:00.502 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-09-21T01:54:00.523 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T01:54:00.566 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T01:54:00.600 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T01:54:00.617 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T01:54:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T01:54:00.664 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T01:54:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T01:54:00.721 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T01:54:00.766 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T01:54:00.768 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T01:54:00.797 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T01:54:00.801 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T01:54:00.849 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T01:54:00.855 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T01:54:00.939 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T01:54:00.976 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T01:54:01.020 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T01:54:01.048 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T01:54:01.120 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T01:54:01.172 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T01:54:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T01:54:01.327 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T01:54:01.453 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:01.478 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:01.570 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:01.599 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:54:01.636 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T01:54:01.638 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T01:54:01.690 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T01:54:01.810 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T01:54:01.864 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T01:54:01.892 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:01.999 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:02.026 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:54:02.091 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:54:02.129 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:54:02.150 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:54:02.218 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T01:54:02.293 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T01:54:02.322 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T01:54:02.353 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:54:02.357 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T01:54:02.403 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T01:54:02.453 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T01:54:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:54:02.612 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T01:54:02.665 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T01:54:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T01:54:02.756 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T01:54:02.799 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:02.829 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T01:54:02.868 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:02.901 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T01:54:02.942 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T01:54:02.986 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T01:54:03.032 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T01:54:03.068 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T01:54:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T01:54:03.153 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T01:54:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T01:54:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T01:54:03.271 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-21T01:54:03.288 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T01:54:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T01:54:03.322 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T01:54:03.364 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T01:54:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T01:54:03.399 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-21T01:54:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T01:54:03.438 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-21T01:54:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T01:54:03.464 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T01:54:03.483 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T01:54:03.492 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-21T01:54:03.506 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T01:54:03.516 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T01:54:03.533 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-21T01:54:03.546 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T01:54:03.552 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T01:54:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-21T01:54:03.579 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T01:54:03.604 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T01:54:03.621 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T01:54:03.623 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-21T01:54:03.683 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T01:54:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T01:54:03.722 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T01:54:03.766 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T01:54:03.846 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T01:54:03.848 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T01:54:03.884 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T01:54:03.928 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T01:54:03.956 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T01:54:03.956 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T01:54:03.984 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T01:54:03.993 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-21T01:54:04.018 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T01:54:04.024 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T01:54:04.057 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T01:54:04.070 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T01:54:04.089 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T01:54:04.100 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:04.119 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T01:54:04.144 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T01:54:04.153 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T01:54:04.187 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T01:54:04.228 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T01:54:04.230 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:04.263 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T01:54:04.268 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:04.286 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T01:54:04.347 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T01:54:04.348 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T01:54:04.361 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:04.390 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:54:04.401 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T01:54:04.420 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-21T01:54:04.433 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T01:54:04.535 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T01:54:04.583 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T01:54:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-21T01:54:04.671 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-21T01:54:04.707 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:04.815 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:04.841 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:54:04.870 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T01:54:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 15/165 2024-09-21T01:54:04.934 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:54:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T01:54:04.963 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-21T01:54:04.997 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T01:54:05.040 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T01:54:05.078 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T01:54:05.121 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T01:54:05.158 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:54:05.285 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T01:54:05.296 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T01:54:05.339 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T01:54:05.348 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T01:54:05.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-21T01:54:05.389 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T01:54:05.430 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T01:54:05.488 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T01:54:05.501 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-21T01:54:05.503 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T01:54:05.536 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T01:54:05.575 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T01:54:05.581 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T01:54:05.608 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T01:54:05.623 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:05.652 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T01:54:05.669 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T01:54:05.698 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T01:54:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:05.734 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T01:54:05.910 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T01:54:05.928 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T01:54:05.936 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-21T01:54:05.948 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T01:54:05.988 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T01:54:06.028 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T01:54:06.029 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T01:54:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-21T01:54:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-21T01:54:06.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-21T01:54:06.175 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T01:54:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-21T01:54:06.213 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T01:54:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-21T01:54:06.255 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T01:54:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-21T01:54:06.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-21T01:54:06.357 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T01:54:06.368 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-21T01:54:06.389 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T01:54:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-21T01:54:06.433 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T01:54:06.474 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T01:54:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-21T01:54:06.499 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T01:54:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-21T01:54:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-21T01:54:06.593 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T01:54:06.608 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-21T01:54:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T01:54:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-21T01:54:06.715 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T01:54:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 34/165 2024-09-21T01:54:06.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T01:54:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 35/165 2024-09-21T01:54:06.796 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T01:54:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 36/165 2024-09-21T01:54:06.800 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T01:54:06.834 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T01:54:06.873 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T01:54:06.874 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-21T01:54:06.913 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T01:54:06.931 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-21T01:54:06.943 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T01:54:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T01:54:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-21T01:54:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T01:54:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T01:54:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T01:54:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-21T01:54:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T01:54:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-21T01:54:07.218 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T01:54:07.264 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T01:54:07.368 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T01:54:07.424 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T01:54:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-21T01:54:07.855 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-21T01:54:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-21T01:54:07.950 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-21T01:54:07.988 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 46/165 2024-09-21T01:54:08.032 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-21T01:54:08.050 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T01:54:08.139 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T01:54:08.143 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T01:54:08.171 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T01:54:08.187 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T01:54:08.192 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-21T01:54:08.204 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T01:54:08.221 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T01:54:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-21T01:54:08.327 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T01:54:08.352 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-21T01:54:08.384 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T01:54:08.415 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-21T01:54:08.429 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T01:54:08.490 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-21T01:54:08.508 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T01:54:08.527 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-21T01:54:08.528 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T01:54:08.541 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T01:54:08.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-21T01:54:08.577 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:08.611 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-21T01:54:08.646 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-21T01:54:08.659 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:08.690 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T01:54:08.698 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-21T01:54:08.725 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T01:54:08.726 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T01:54:08.747 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T01:54:08.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-21T01:54:08.832 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T01:54:08.891 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-21T01:54:08.933 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-21T01:54:08.975 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-21T01:54:09.080 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-21T01:54:09.125 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-21T01:54:09.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-21T01:54:09.202 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-21T01:54:09.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T01:54:09.226 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-21T01:54:09.247 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:09.309 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:09.329 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-21T01:54:09.340 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T01:54:09.458 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-21T01:54:09.476 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T01:54:09.493 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-21T01:54:09.521 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-21T01:54:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-21T01:54:09.585 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T01:54:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-21T01:54:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-21T01:54:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-21T01:54:09.724 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-21T01:54:09.757 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-21T01:54:09.799 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-21T01:54:09.833 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-21T01:54:09.925 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-21T01:54:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-21T01:54:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-21T01:54:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-21T01:54:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-21T01:54:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T01:54:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T01:54:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T01:54:10.988 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T01:54:11.321 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T01:54:11.337 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-21T01:54:11.380 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-21T01:54:11.384 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:11.414 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-21T01:54:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T01:54:11.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-21T01:54:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T01:54:11.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-21T01:54:11.647 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-21T01:54:11.647 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:11.647 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-21T01:54:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-21T01:54:12.031 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T01:54:12.049 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-21T01:54:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:12.080 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-21T01:54:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:12.168 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-21T01:54:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T01:54:12.224 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:12.256 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:12.330 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:12.331 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T01:54:12.331 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T01:54:12.331 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:12.366 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:12.975 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-21T01:54:13.091 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-21T01:54:14.403 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-21T01:54:14.484 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-21T01:54:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-21T01:54:14.518 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-21T01:54:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-21T01:54:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 102/165 2024-09-21T01:54:15.056 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 103/165 2024-09-21T01:54:15.092 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-21T01:54:15.103 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:15.146 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:15.232 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:15.232 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T01:54:15.232 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T01:54:15.232 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:15.281 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-21T01:54:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 106/165 2024-09-21T01:54:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 107/165 2024-09-21T01:54:18.510 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:18.544 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:19.156 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 108/165 2024-09-21T01:54:19.207 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:19.300 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 109/165 2024-09-21T01:54:19.301 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T01:54:19.301 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T01:54:19.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:19.342 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:24.378 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:24.378 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-09-21T01:54:24.378 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-09-21T01:54:24.379 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:24.551 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:24.567 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:24.567 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:24.567 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T01:54:24.567 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:26.162 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:26.189 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T01:54:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:26.190 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:26.413 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:26.436 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:26.436 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:26.436 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T01:54:26.436 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:26.436 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:26.437 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:26.883 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:26.883 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-09-21T01:54:26.884 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:26.913 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:26.937 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:27.043 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T01:54:27.066 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:27.082 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:27.083 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T01:54:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:27.099 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T01:54:27.155 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T01:54:27.341 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T01:54:27.397 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T01:54:27.474 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T01:54:27.563 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T01:54:27.709 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T01:54:27.779 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T01:54:27.807 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T01:54:28.259 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:28.289 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:28.321 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T01:54:28.387 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T01:54:28.412 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:28.661 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:28.687 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:28.687 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:28.688 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T01:54:28.688 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:28.688 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:28.688 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:28.920 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:28.943 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:29.444 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:29.444 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T01:54:29.444 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:29.444 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:29.551 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T01:54:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T01:54:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T01:54:29.848 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T01:54:29.896 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T01:54:29.974 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T01:54:30.063 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T01:54:30.210 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T01:54:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T01:54:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T01:54:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T01:54:30.757 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:30.796 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T01:54:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T01:54:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:31.590 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:31.660 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 110/165 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-09-21T01:54:31.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:31.964 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:31.981 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 111/165 2024-09-21T01:54:31.981 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:31.981 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T01:54:31.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:32.766 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:32.944 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T01:54:32.979 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:33.003 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:33.054 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:33.066 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T01:54:33.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T01:54:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 112/165 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:54:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:33.901 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 113/165 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:54:33.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:34.196 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T01:54:34.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:34.247 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:34.247 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:34.247 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T01:54:34.247 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:34.247 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:34.248 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:34.291 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:34.314 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:34.314 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:34.314 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T01:54:34.314 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:34.366 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:34.389 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:34.389 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:34.389 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T01:54:34.390 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:34.390 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:34.390 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 114/165 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:54:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:34.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-21T01:54:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-21T01:54:34.680 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-21T01:54:34.717 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:34.741 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:34.882 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-21T01:54:34.929 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-21T01:54:35.009 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-21T01:54:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-21T01:54:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-21T01:54:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-21T01:54:35.298 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-21T01:54:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-21T01:54:35.461 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T01:54:35.494 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:35.519 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:35.570 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:35.582 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T01:54:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T01:54:35.799 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:35.837 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 126/165 2024-09-21T01:54:35.878 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 127/165 2024-09-21T01:54:35.947 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 128/165 2024-09-21T01:54:35.985 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:36.723 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T01:54:36.756 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:36.779 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:36.823 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:36.846 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:36.846 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T01:54:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:36.898 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:36.922 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:37.249 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:37.273 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:38.749 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T01:54:38.796 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T01:54:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T01:54:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T01:54:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T01:54:39.113 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T01:54:39.167 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T01:54:39.198 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T01:54:39.227 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T01:54:39.273 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T01:54:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 129/165 2024-09-21T01:54:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:39.857 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:39.896 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:39.916 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 130/165 2024-09-21T01:54:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 131/165 2024-09-21T01:54:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T01:54:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 132/165 2024-09-21T01:54:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 133/165 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:54:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:40.894 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T01:54:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 134/165 2024-09-21T01:54:40.992 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T01:54:41.020 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T01:54:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 135/165 2024-09-21T01:54:41.046 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T01:54:41.072 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T01:54:41.072 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 136/165 2024-09-21T01:54:41.156 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:41.156 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:41.240 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:41.264 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T01:54:41.270 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:41.270 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:41.295 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T01:54:41.336 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T01:54:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T01:54:41.425 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T01:54:41.441 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:41.441 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:41.453 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T01:54:41.534 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:41.607 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T01:54:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T01:54:41.738 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T01:54:41.768 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T01:54:41.814 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T01:54:42.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-21T01:54:42.139 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 138/165 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:54:42.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:42.207 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:42.231 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432 139/165 2024-09-21T01:54:42.231 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:42.231 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T01:54:42.231 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:42.282 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:42.307 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 140/165 2024-09-21T01:54:42.349 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:42.350 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-21T01:54:42.350 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:42.350 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-21T01:54:42.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 141/165 2024-09-21T01:54:42.675 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:54:42.675 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T01:54:42.675 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:42.675 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:54:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:43.316 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T01:54:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T01:54:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T01:54:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T01:54:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T01:54:43.602 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T01:54:43.636 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T01:54:43.636 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:43.712 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:43.713 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:43.796 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:43.821 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T01:54:43.827 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:43.827 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:44.006 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:44.007 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:46.922 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 142/165 2024-09-21T01:54:46.963 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 143/165 2024-09-21T01:54:47.012 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 144/165 2024-09-21T01:54:47.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-21T01:54:47.079 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-21T01:54:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 147/165 2024-09-21T01:54:47.360 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-21T01:54:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 149/165 2024-09-21T01:54:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-21T01:54:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-21T01:54:47.936 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:47.936 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T01:54:47.936 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T01:54:47.937 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T01:54:47.938 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T01:54:47.939 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T01:54:47.940 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T01:54:47.941 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T01:54:47.942 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T01:54:47.943 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T01:54:47.944 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T01:54:47.945 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T01:54:47.946 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T01:54:47.947 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T01:54:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-21T01:54:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 153/165 2024-09-21T01:54:49.178 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-21T01:54:49.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-21T01:54:49.322 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-21T01:54:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-21T01:54:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-21T01:54:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-21T01:54:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-21T01:54:49.500 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:49.584 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-21T01:54:49.609 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-21T01:54:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:49.615 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-21T01:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:49.879 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-21T01:54:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:50.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T01:54:50.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T01:54:50.440 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T01:54:50.441 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T01:54:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T01:54:50.443 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T01:54:50.444 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T01:54:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T01:54:50.446 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T01:54:50.447 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T01:54:50.448 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T01:54:50.449 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T01:54:50.580 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:50.580 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:50.580 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-09-21T01:54:50.580 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:50.580 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.581 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.582 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T01:54:50.583 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T01:54:50.584 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T01:54:50.585 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T01:54:50.586 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T01:54:50.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T01:54:50.588 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T01:54:50.589 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T01:54:50.590 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T01:54:50.591 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:54:50.592 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-09-21T01:54:51.014 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:53.338 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-09-21T01:54:53.339 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.340 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T01:54:53.341 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.342 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T01:54:53.343 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T01:54:53.344 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T01:54:53.345 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T01:54:53.346 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T01:54:53.347 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T01:54:53.348 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T01:54:53.349 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T01:54:53.350 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T01:54:53.350 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:54:53.350 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-09-21T01:54:53.793 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 3/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 4/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432 5/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 6/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 7/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 8/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 9/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 10/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 11/165 2024-09-21T01:54:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 12/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 13/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 14/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_ 15/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 16/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 17/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_6 18/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86 19/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 20/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el 21/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9. 22/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_6 23/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 24/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 25/165 2024-09-21T01:54:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 26/165 2024-09-21T01:54:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 27/165 2024-09-21T01:54:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 28/165 2024-09-21T01:54:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 29/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218. 30/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noa 31/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.n 32/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g184 33/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el 34/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 35/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.e 36/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 37/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 38/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 39/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-21T01:54:56.424 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-21T01:54:56.425 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-21T01:54:56.426 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-21T01:54:56.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-21T01:54:56.428 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-21T01:54:56.429 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-21T01:54:56.430 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-21T01:54:56.431 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 150/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 152/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-21T01:54:56.432 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-21T01:54:56.433 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:59.328 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.329 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.330 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T01:54:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T01:54:59.332 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.16-1.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-21T01:54:59.333 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T01:54:59.334 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T01:54:59.335 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T01:54:59.336 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T01:54:59.337 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T01:54:59.338 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T01:54:59.339 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-21T01:54:59.340 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:54:59.340 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-21T01:54:59.899 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:59.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:00.046 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T01:55:00.070 INFO:teuthology.orchestra.run.smithi016.stdout:19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.071 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.071 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T01:55:00.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:00.215 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T01:55:00.241 INFO:teuthology.orchestra.run.smithi123.stdout:19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.241 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.241 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T01:55:00.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:00.385 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T01:55:00.412 INFO:teuthology.orchestra.run.smithi161.stdout:19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.412 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T01:55:00.412 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T01:55:00.413 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T01:55:00.414 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:55:00.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T01:55:00.447 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:55:00.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T01:55:00.479 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:55:00.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T01:55:00.510 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T01:55:00.510 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:55:00.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T01:55:00.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T01:55:00.608 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:55:00.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T01:55:00.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T01:55:00.702 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:55:00.702 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T01:55:00.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T01:55:00.801 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T01:55:00.801 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:55:00.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T01:55:00.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T01:55:00.897 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:55:00.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T01:55:00.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T01:55:00.991 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:55:00.991 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T01:55:01.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T01:55:01.089 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T01:55:01.089 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:55:01.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T01:55:01.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T01:55:01.184 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:55:01.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T01:55:01.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T01:55:01.279 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:55:01.280 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T01:55:01.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T01:55:01.378 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T01:55:01.466 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug', 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'mds': {'mds_bal_replicate_threshold': 1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'mgr-modules': ['snap_schedule'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'subvols': {'create': 2}} 2024-09-21T01:55:01.466 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:01.467 INFO:tasks.cephadm:Cluster fsid is 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:55:01.467 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T01:55:01.467 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.123', 'mon.c': '172.21.15.161'} 2024-09-21T01:55:01.467 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-09-21T01:55:01.467 INFO:tasks.cephadm:First mgr is x 2024-09-21T01:55:01.467 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T01:55:01.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-09-21T01:55:01.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-09-21T01:55:01.525 DEBUG:teuthology.orchestra.run.smithi161:> sudo hostname $(hostname -s) 2024-09-21T01:55:01.555 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T01:55:01.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:01.699 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 00:04:17.484514', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 22:54:16.499090', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T01:55:01.840 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240920.212106-debug, sha1 1843221895adccfb3b692060251dd6a0183bc410 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:55:01.841 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T01:55:01.841 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T01:55:01.841 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:02.373 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 01:55 /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:02.373 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:02.911 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 01:55 /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:02.911 DEBUG:teuthology.orchestra.run.smithi161:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:03.464 INFO:teuthology.orchestra.run.smithi161.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 01:55 /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:03.465 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:03.485 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:03.504 DEBUG:teuthology.orchestra.run.smithi161:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T01:55:03.540 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 on all hosts... 2024-09-21T01:55:03.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T01:55:03.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T01:55:03.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T01:55:05.089 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:55:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:55:05.286 INFO:teuthology.orchestra.run.smithi161.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:55:35.225 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-21T01:55:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:55:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T01:55:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-09-21T01:55:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T01:55:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-21T01:55:35.226 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-21T01:55:36.954 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T01:55:36.954 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:55:36.954 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T01:55:36.955 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-09-21T01:55:36.955 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T01:55:36.955 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T01:55:36.955 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T01:55:38.280 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-21T01:55:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:55:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T01:55:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "repo_digests": [ 2024-09-21T01:55:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T01:55:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-21T01:55:38.281 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-21T01:55:38.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-09-21T01:55:38.339 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-09-21T01:55:38.373 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2024-09-21T01:55:38.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-09-21T01:55:38.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-09-21T01:55:38.456 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /etc/ceph 2024-09-21T01:55:38.481 INFO:tasks.cephadm:Writing seed config... 2024-09-21T01:55:38.481 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-21T01:55:38.482 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T01:55:38.483 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-09-21T01:55:38.484 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:55:38.484 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T01:55:38.502 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8372ca74-77bc-11ef-baf5-efdc52797490 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_bal_replicate_threshold = 1 2024-09-21T01:55:38.502 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service 2024-09-21T01:55:38.545 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x.service 2024-09-21T01:55:38.587 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T01:55:38.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 -v bootstrap --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:55:38.805 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:55:38.805 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', '-v', 'bootstrap', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-09-21T01:55:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T01:55:38.805 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-09-21T01:55:38.830 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-21T01:55:38.831 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-09-21T01:55:38.831 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-09-21T01:55:38.838 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T01:55:38.839 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T01:55:38.845 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T01:55:38.845 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:55:38.852 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-21T01:55:38.858 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-21T01:55:38.859 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-21T01:55:38.859 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-09-21T01:55:38.881 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.2.2 2024-09-21T01:55:38.882 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T01:55:38.882 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-09-21T01:55:38.882 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-09-21T01:55:38.888 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T01:55:38.888 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T01:55:38.895 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T01:55:38.895 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:55:38.902 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-09-21T01:55:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-09-21T01:55:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-09-21T01:55:38.908 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 139827236690576 on /run/cephadm/8372ca74-77bc-11ef-baf5-efdc52797490.lock 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 139827236690576 acquired on /run/cephadm/8372ca74-77bc-11ef-baf5-efdc52797490.lock 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-09-21T01:55:38.909 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-21T01:55:38.913 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-09-21T01:55:38.913 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-09-21T01:55:38.915 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T01:55:38.916 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T01:55:38.916 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1748sec hoplimit 64 pref medium 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-21T01:55:38.918 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T01:55:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-21T01:55:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-09-21T01:55:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T01:55:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T01:55:38.921 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:55:39.398 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T01:55:39.398 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:55:39.398 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T01:55:39.399 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:8425bc7408d91b180e7fdc62f23e7ae6603757f4f36e9969a4eaf9fd9416b764 2024-09-21T01:55:39.399 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T01:55:39.399 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T01:55:39.399 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T01:55:40.362 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T01:55:40.362 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T01:55:40.362 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T01:55:41.351 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-09-21T01:55:41.351 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-09-21T01:55:42.231 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCdJ+5md7GQJxAABepW7aG6Gj0oufWMCZBPLA== 2024-09-21T01:55:51.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCgJ+5me43hGhAA9XTVAdWnxcfS9cSa8JcGGw== 2024-09-21T01:55:56.245 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQCqJ+5mh9AaChAA1f3Pyhm7K8Ycd1FUFg8yPw== 2024-09-21T01:55:56.246 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-09-21T01:55:57.651 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:55:57.652 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-09-21T01:55:58.533 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-09-21T01:55:58.740 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T01:55:59.043 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T01:55:59.226 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target. 2024-09-21T01:55:59.226 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target. 2024-09-21T01:55:59.489 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a 2024-09-21T01:55:59.490 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-21T01:55:59.659 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target.wants/ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490@.service. 2024-09-21T01:56:00.190 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:56:00.190 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-09-21T01:56:00.198 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-09-21T01:56:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:01 smithi016 ceph-mon[30877]: from='client.? 172.21.15.16:0/1739472323' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T01:56:01.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.547578s) 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T01:56:01.978 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T01:56:01.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T01:56:01.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T01:56:01.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:01.979 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-09-21T01:56:01.979 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T01:56:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:02 smithi016 ceph-mon[30877]: from='client.? 172.21.15.16:0/707183414' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T01:56:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:02 smithi016 ceph-mon[30877]: from='client.? 172.21.15.16:0/707183414' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T01:56:02.998 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T01:56:02.999 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-09-21T01:56:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:03 smithi016 ceph-mon[30877]: from='client.? 172.21.15.16:0/4001336311' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:56:03.920 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-09-21T01:56:04.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:03 smithi016 systemd[1]: Stopping Ceph mon.a for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T01:56:04.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a[30873]: 2024-09-21T01:56:04.211+0000 7fe08bf2a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:56:04.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a[30873]: 2024-09-21T01:56:04.211+0000 7fe08bf2a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T01:56:04.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:04 smithi016 podman[31119]: 2024-09-21 01:56:04.273262276 +0000 UTC m=+0.177316560 container died 9cb4110cbf14473e561283122d946ff2f50922ab1cf60af32b228b41e64310bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:56:05.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:04 smithi016 podman[31119]: 2024-09-21 01:56:04.736663004 +0000 UTC m=+0.640717276 container cleanup 9cb4110cbf14473e561283122d946ff2f50922ab1cf60af32b228b41e64310bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T01:56:05.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:04 smithi016 bash[31119]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a 2024-09-21T01:56:05.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 podman[31131]: 2024-09-21 01:56:05.004424859 +0000 UTC m=+0.726574286 container remove 9cb4110cbf14473e561283122d946ff2f50922ab1cf60af32b228b41e64310bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T01:56:05.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 systemd[1]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T01:56:05.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 systemd[1]: Stopped Ceph mon.a for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:56:05.381 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 systemd[1]: Starting Ceph mon.a for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T01:56:05.628 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T01:56:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 podman[31225]: 2024-09-21 01:56:05.28402462 +0000 UTC m=+0.016401697 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:56:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 podman[31225]: 2024-09-21 01:56:05.388469226 +0000 UTC m=+0.120846289 container create e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T01:56:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 podman[31225]: 2024-09-21 01:56:05.566249947 +0000 UTC m=+0.298627019 container init e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:56:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 podman[31225]: 2024-09-21 01:56:05.569388456 +0000 UTC m=+0.301765522 container start e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T01:56:05.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: pidfile_write: ignore empty --pid-file 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: load: jerasure load: lrc 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: RocksDB version: 7.9.2 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Git sha 0 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: DB SUMMARY 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: DB Session ID: F1P0I987C20NP7DKTHRP 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: CURRENT file: CURRENT 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T01:56:05.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T01:56:05.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94284 ; 2024-09-21T01:56:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.error_if_exists: 0 2024-09-21T01:56:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.create_if_missing: 0 2024-09-21T01:56:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.paranoid_checks: 1 2024-09-21T01:56:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T01:56:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.env: 0x55893250ed60 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.info_log: 0x558936f69360 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.statistics: (nil) 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.use_fsync: 0 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_log_file_size: 0 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T01:56:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_fallocate: 1 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.use_direct_reads: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.db_log_dir: 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.wal_dir: 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T01:56:05.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.write_buffer_manager: 0x558936f6dd60 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.unordered_write: 0 2024-09-21T01:56:05.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.row_cache: None 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.wal_filter: None 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.two_write_queues: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.wal_compression: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.atomic_flush: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T01:56:05.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.log_readahead_size: 0 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_background_jobs: 2 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_background_compactions: -1 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_subcompactions: 1 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T01:56:05.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_open_files: -1 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_background_flushes: -1 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Compression algorithms supported: 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kZSTD supported: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kXpressCompression supported: 0 2024-09-21T01:56:05.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kBZip2Compression supported: 0 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kLZ4Compression supported: 1 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kZlibCompression supported: 1 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: kSnappyCompression supported: 1 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.merge_operator: 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_filter: None 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T01:56:05.669 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558936f68fa0) 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x558932f2d610 2024-09-21T01:56:05.670 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-09-21T01:56:05.671 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression: NoCompression 2024-09-21T01:56:05.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.num_levels: 7 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:56:05.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T01:56:05.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T01:56:05.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.inplace_update_support: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.bloom_locality: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.max_successive_merges: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T01:56:05.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.ttl: 2592000 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enable_blob_files: false 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.min_blob_size: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T01:56:05.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 218d99df-8c8f-4298-a029-0f5c5e0a97f4 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883765606208, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883765607384, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90588, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 281, "table_properties": {"data_size": 88629, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11462, "raw_average_key_size": 47, "raw_value_size": 82109, "raw_average_value_size": 342, "num_data_blocks": 12, "num_entries": 240, "num_filter_entries": 240, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726883765, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "218d99df-8c8f-4298-a029-0f5c5e0a97f4", "db_session_id": "F1P0I987C20NP7DKTHRP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883765607459, "job": 1, "event": "recovery_finished"} 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558936f8ca80 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: DB pointer 0x558937d32000 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T01:56:05.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 90.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:56:05.680 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 90.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 118.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T01:56:05.681 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 24.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 24.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x558932f2d610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???) e1 preinit fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).mds e1 new map 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).mds e1 print_map 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-09-21T01:56:05.682 INFO:journalctl@ceph.mon.a.smithi016.stdout: btime 2024-09-21T01:56:00:153260+0000 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 bash[31225]: e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 2024-09-21T01:56:05.683 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 systemd[1]: Started Ceph mon.a for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: monmap epoch 1 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: last_changed 2024-09-21T01:55:56.591199+0000 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:56:06.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: min_mon_release 19 (squid) 2024-09-21T01:56:06.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: election_strategy: 1 2024-09-21T01:56:06.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:56:06.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: fsmap 2024-09-21T01:56:06.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T01:56:06.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:05 smithi016 ceph-mon[31245]: mgrmap e1: no daemons active 2024-09-21T01:56:06.968 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T01:56:06.989 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:56:06.989 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-09-21T01:56:06.990 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T01:56:06.990 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T01:56:07.256 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x 2024-09-21T01:56:07.256 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x.service: Unit ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x.service not loaded. 2024-09-21T01:56:07.417 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490.target.wants/ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x.service → /etc/systemd/system/ceph-8372ca74-77bc-11ef-baf5-efdc52797490@.service. 2024-09-21T01:56:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:07 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3512172174' entity='client.admin' 2024-09-21T01:56:07.658 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 systemd[1]: Starting Ceph mgr.x for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T01:56:07.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 podman[31455]: 2024-09-21 01:56:07.658321794 +0000 UTC m=+0.111437133 container create 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:56:07.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 podman[31455]: 2024-09-21 01:56:07.574441639 +0000 UTC m=+0.027556947 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:56:07.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 podman[31455]: 2024-09-21 01:56:07.869449821 +0000 UTC m=+0.322565128 container init 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T01:56:07.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 podman[31455]: 2024-09-21 01:56:07.872208971 +0000 UTC m=+0.325324283 container start 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:56:07.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 bash[31455]: 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 2024-09-21T01:56:07.961 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:56:07.961 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-21T01:56:07.970 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:56:07.970 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:56:07.970 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-21T01:56:07.970 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T01:56:07.981 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:56:07.981 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-09-21T01:56:07.989 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-09-21T01:56:08.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:07 smithi016 systemd[1]: Started Ceph mgr.x for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:56:08.792 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:08 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2657285506' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:56:09.109 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:09.109 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:56:09.110 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:09.111 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.113 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:09.114 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:56:00:153260+0000", 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:56:09.115 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:56:00.154084+0000", 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-09-21T01:56:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2334601270' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:56:12.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:56:12.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:56:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:56:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:56:12.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:56:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:56:00:153260+0000", 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:56:12.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:56:12.213 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:56:00.154084+0000", 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:12.214 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: Activating manager daemon x 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: mgrmap e2: x(active, starting, since 0.00341102s) 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:56:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: Manager daemon x is now available 2024-09-21T01:56:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:56:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:56:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' 2024-09-21T01:56:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' 2024-09-21T01:56:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:13 smithi016 ceph-mon[31245]: from='mgr.14100 172.21.15.16:0/2510061053' entity='mgr.x' 2024-09-21T01:56:15.464 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:15 smithi016 ceph-mon[31245]: mgrmap e3: x(active, since 1.00863s) 2024-09-21T01:56:15.464 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:15 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3977309322' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:56:15.468 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:15.468 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:15.468 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T01:56:15.468 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:56:15.470 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:56:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:56:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:56:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:56:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:56:00:153260+0000", 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:56:15.474 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:56:00.154084+0000", 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:56:15.475 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:56:15.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:56:15.476 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:15.476 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-09-21T01:56:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:16 smithi016 ceph-mon[31245]: mgrmap e4: x(active, since 2s) 2024-09-21T01:56:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:16 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1977221661' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T01:56:16.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:16.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-21T01:56:16.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-21T01:56:16.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T01:56:16.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T01:56:16.610 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-09-21T01:56:17.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/173056756' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T01:56:17.601 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:17 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setuser ceph since I am not root 2024-09-21T01:56:17.601 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:17 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setgroup ceph since I am not root 2024-09-21T01:56:18.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:18 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/173056756' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T01:56:18.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:18 smithi016 ceph-mon[31245]: mgrmap e5: x(active, since 4s) 2024-09-21T01:56:19.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:19.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T01:56:19.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:56:19.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T01:56:19.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T01:56:19.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:19.154 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-21T01:56:19.154 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-09-21T01:56:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:19 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2405582816' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: Active manager daemon x restarted 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: Activating manager daemon x 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: mgrmap e6: x(active, starting, since 0.00350364s) 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:56:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:22 smithi016 ceph-mon[31245]: Manager daemon x is now available 2024-09-21T01:56:24.091 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:24.092 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T01:56:24.092 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T01:56:24.092 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:24.092 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-09-21T01:56:24.092 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying orchestrator module is enabled... 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: Found migration_current of "None". Setting to last migration. 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:56:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:56:24.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T01:56:24.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: mgrmap e7: x(active, since 1.00668s) 2024-09-21T01:56:24.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:24 smithi016 ceph-mon[31245]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T01:56:25.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3673748330' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T01:56:25.861 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:56:25.683+0000 7ffb15b41640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:01:56:25] ENGINE Error in HTTPServer.serve 2024-09-21T01:56:25.861 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Traceback (most recent call last): 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._connections.run(self.expiration_interval) 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._run(expiration_interval) 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s = self.context.wrap_socket( 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: return self.sslsocket_class._create( 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:56:25.862 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self.do_handshake() 2024-09-21T01:56:25.863 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:56:25.863 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._sslobj.do_handshake() 2024-09-21T01:56:25.863 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:56:25.863 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:56:26.123 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T01:56:26.123 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:25] ENGINE Bus STARTING 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3673748330' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: mgrmap e8: x(active, since 2s) 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:25] ENGINE Serving on http://172.21.15.16:8765 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:25] ENGINE Serving on https://172.21.15.16:7150 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:25] ENGINE Bus STARTED 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:26 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:25] ENGINE Error in HTTPServer.serve 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-21T01:56:26.774 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:56:26.775 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-21T01:56:26.776 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:56:26.776 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-21T01:56:26.776 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:56:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:27 smithi016 ceph-mon[31245]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:27 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:27 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:28.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T01:56:28.544 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-09-21T01:56:28.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:28 smithi016 ceph-mon[31245]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Generating public/private rsa key pair. 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Your identification has been saved in /tmp/tmpildtusa6/key 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Your public key has been saved in /tmp/tmpildtusa6/key.pub 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: The key fingerprint is: 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: SHA256:7rsR8gClCjzEcczvL4FbDhU7rvDMDCg5izjImBzDKOQ ceph-8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: The key's randomart image is: 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: +---[RSA 3072]----+ 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |.o+. . | 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |o..o + | 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |.o + o | 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: | o.. * | 2024-09-21T01:56:29.761 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |*.. = + S | 2024-09-21T01:56:29.762 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |BE o = = . | 2024-09-21T01:56:29.762 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |X+X * o + | 2024-09-21T01:56:29.762 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: |Oo B o o . | 2024-09-21T01:56:29.762 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: | . . +o | 2024-09-21T01:56:29.762 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: +----[SHA256]-----+ 2024-09-21T01:56:30.116 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-mon[31245]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:30.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-mon[31245]: Generating ssh key... 2024-09-21T01:56:30.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:30.117 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:29 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:31.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:56:31.159 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:56:31.159 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T01:56:31.166 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-09-21T01:56:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:31 smithi016 ceph-mon[31245]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:32.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:32 smithi016 ceph-mon[31245]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:33 smithi016 ceph-mon[31245]: Deploying cephadm binary to smithi016 2024-09-21T01:56:36.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-09-21T01:56:36.167 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-09-21T01:56:36.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:36 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:36 smithi016 ceph-mon[31245]: Added host smithi016 2024-09-21T01:56:36.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:36 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:38.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:37 smithi016 ceph-mon[31245]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:38.154 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:37 smithi016 ceph-mon[31245]: Saving service mon spec with placement count:5 2024-09-21T01:56:38.154 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:37 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:38.154 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:37 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:38.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T01:56:38.415 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-09-21T01:56:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:39 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:39 smithi016 ceph-mon[31245]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:39 smithi016 ceph-mon[31245]: Saving service mgr spec with placement count:2 2024-09-21T01:56:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:39 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:40.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T01:56:41.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:40 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:41.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:40 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3974542523' entity='client.admin' 2024-09-21T01:56:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:42 smithi016 ceph-mon[31245]: from='mgr.14118 172.21.15.16:0/3191849031' entity='mgr.x' 2024-09-21T01:56:42.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:42 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1605382177' entity='client.admin' 2024-09-21T01:56:42.666 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-09-21T01:56:43.812 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:43 smithi016 ceph-mon[31245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:56:44.890 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:44 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/474603712' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T01:56:44.890 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setuser ceph since I am not root 2024-09-21T01:56:44.891 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setgroup ceph since I am not root 2024-09-21T01:56:45.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:45 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/474603712' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T01:56:45.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:45 smithi016 ceph-mon[31245]: mgrmap e9: x(active, since 22s) 2024-09-21T01:56:46.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:46.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T01:56:46.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:56:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-21T01:56:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T01:56:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-09-21T01:56:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-09-21T01:56:46.783 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:46 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1866259614' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: Active manager daemon x restarted 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: Activating manager daemon x 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: mgrmap e10: x(active, starting, since 0.0044141s) 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:56:50.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:49 smithi016 ceph-mon[31245]: Manager daemon x is now available 2024-09-21T01:56:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:50 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:56:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:50 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:56:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:50 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:56:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:50 smithi016 ceph-mon[31245]: mgrmap e11: x(active, since 1.00699s) 2024-09-21T01:56:51.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:56:51.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T01:56:51.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T01:56:51.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:56:51.483 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-09-21T01:56:51.483 INFO:teuthology.orchestra.run.smithi016.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T01:56:52.143 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-mon[31245]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-mon[31245]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:56:51.847+0000 7f52d4abd640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:01:56:51] ENGINE Error in HTTPServer.serve 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Traceback (most recent call last): 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._connections.run(self.expiration_interval) 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._run(expiration_interval) 2024-09-21T01:56:52.144 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s = self.context.wrap_socket( 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: return self.sslsocket_class._create( 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:56:52.145 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self.do_handshake() 2024-09-21T01:56:52.146 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:56:52.146 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._sslobj.do_handshake() 2024-09-21T01:56:52.146 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:56:52.146 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:56:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:56:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:51] ENGINE Bus STARTING 2024-09-21T01:56:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:51] ENGINE Serving on https://172.21.15.16:7150 2024-09-21T01:56:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:51] ENGINE Error in HTTPServer.serve 2024-09-21T01:56:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-21T01:56:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-21T01:56:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: mgrmap e12: x(active, since 2s) 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:51] ENGINE Serving on http://172.21.15.16:8765 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: [21/Sep/2024:01:56:51] ENGINE Bus STARTED 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:53.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:52 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:56:53.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIURtoLyAjP2L33GX9SceHEKVlrVkgwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTAxNTY1M1oXDTM0\nMDkyMjAxNTY1M1owFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjE2MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAoPvQytmCa4G6L7/XtUV8u9Q9plJYfxtWNR3O\nESNZ34paBNRJbE2ionBTbvKM5c6VojyaraZuVKcf8QV8bDIOaGbeEnFoQDNXZ+Hq\nXNPXBKnUmkZPclulerE64kwxJjJP/xq9mblxl3R6VR4sKFDsB/YGwQ2UIS85vOHC\nCXB5omPuS+VGm6unuc2D8U7RAmVLmPOIkcwOw4HTbCqLqt53lreLrS+7k9raZAGz\n545FYTdlWwv7ZJm+5j7m26ww65UCo5XaU6rWT3g5zuyrM89rKxMTwheRAMOOIppK\nPYMXsVNr30DS70W5uQQf0N2SWjX01dv01QkvxooPq6Q6l0y4PnNE8xqenQ25r+Be\nkApJ7NLyFmwKKX54pmtPhgeefr4Os1ZcX7duOTqkGm45xyGT0J1AnohcZnqYcpDo\nmmS9xI1Ayh8Ks/1NEjPLM6istIK6hu8y7DEQ6261XmMJnueJAmp/kyBLX/pTQokz\n0rWhdcFlIq9hVlBCXl29lp6mbftseNtKfyipc1I4lA3S+Ra/2T9mtle6fVWPnxjM\n7Toa3ialNlOSgsih79Dh2I9r39MEAD/+rVbwe9Fzg+8B2CFkCotyleL4LuxltTg/\nfNtArpKkS8QDbB9/2UxfYJgnyaFDf08+c/7LUPNZnxjM3CCciu6GFs8Y5CawmNp4\nAG7sMIMCAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDE2ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPEDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQCf\nCrJFEVME/FPGyWlioGin3gItYkwwuX7NhrPrMYUaSz43Zd54FJ4P4HmgEUfKYD19\n83PGMxVt+9eeESmd8FJPmecz0PdpMniKlLgJWmewPHS7afOUqAf0OH6wbmTNK/t3\nExuyHN+B58AGATV+8/NGXia74rBH3IhF9sU7RlGNmmBtf229wjgDLGPLoqCdssO+\nD+b5ihMKBC/T+JuG3e3vsI9vy7rDZvCVXKuu/LE7bzGljM0tD34WrzVi85zvgIrm\nejraUZByAlKaC4O/0SitQyjSwcGzs+jauSWvY4KkuQqAceriwCK8T9uoNtibhHYc\nsKqBrVSmHsD0YCxogUdA/0DY5VUWHD1rTbsphgFmLh5sEUS4fiGu21zBmpZE0JmL\ngfHricJQkiSdHY5f2zcCVv+OEa/h0RD3fq8r9powMh9AvUD1dOCBVGR+NhvB39ta\nIIQXVxNMLKSW5GmMWt8wiq2jGvbMLgpwjYx9LFLzCnEJNc4USwUBezkd5SmrsDl0\nSs9D00IF6ESXSNCdxruQZRY1LJZ90te9xSgwPbW/gshiXaBEmnbTLwjL7qKt+tph\n3KZWGSWGmHMWMEiJRXwNoL17tkMURD5Vg8RXBKsuVUGUVTfOAIqwVaxKs8CfxVJt\nepGSPQ4K/CskhL9C7rfD1lQmdFI5ibohYY1PIDjDYQ==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJJwIBAAKCAgEAoPvQytmCa4G6L7/XtUV8u9Q9plJYfxtWNR3OESNZ34paBNRJ\nbE2ionBTbvKM5c6VojyaraZuVKcf8QV8bDIOaGbeEnFoQDNXZ+HqXNPXBKnUmkZP\nclulerE64kwxJjJP/xq9mblxl3R6VR4sKFDsB/YGwQ2UIS85vOHCCXB5omPuS+VG\nm6unuc2D8U7RAmVLmPOIkcwOw4HTbCqLqt53lreLrS+7k9raZAGz545FYTdlWwv7\nZJm+5j7m26ww65UCo5XaU6rWT3g5zuyrM89rKxMTwheRAMOOIppKPYMXsVNr30DS\n70W5uQQf0N2SWjX01dv01QkvxooPq6Q6l0y4PnNE8xqenQ25r+BekApJ7NLyFmwK\nKX54pmtPhgeefr4Os1ZcX7duOTqkGm45xyGT0J1AnohcZnqYcpDommS9xI1Ayh8K\ns/1NEjPLM6istIK6hu8y7DEQ6261XmMJnueJAmp/kyBLX/pTQokz0rWhdcFlIq9h\nVlBCXl29lp6mbftseNtKfyipc1I4lA3S+Ra/2T9mtle6fVWPnxjM7Toa3ialNlOS\ngsih79Dh2I9r39MEAD/+rVbwe9Fzg+8B2CFkCotyleL4LuxltTg/fNtArpKkS8QD\nbB9/2UxfYJgnyaFDf08+c/7LUPNZnxjM3CCciu6GFs8Y5CawmNp4AG7sMIMCAwEA\nAQKCAgABJO5ogAr1xv/ITBr1PmUIaWtLbdq+2Iu7J+w+4fCcbqgnevd5+MP3AfO4\niP/TDa4G77BXxfmTH3b80xb90TiZ/wqFY19O1C6YXe+OW7EeZt33KhhxeHugzMwI\nL2t1WTaJWoAjghZvxv5W8YvLI5Fx7bVBlQj1rWzBzhxUzdhc0kyvfGs35bdKUk7Q\nSHOi3+PILAmko6iCij8y+HXIBU8goqBbCZ5R4Dnc1nLtapeip8P61YciwWxXyEsC\nymK0oflZU/zadbnn6bOw0qzVhZwTkSNQULORUlKF0MabUP26ScdYAxFyGEhIK40r\nAmWCak9vCmJfS0fsJ4S8qcJYMKLp8n29JwyUSpq2Y/xeYAfogbfq1IvL4j91WHgk\nmk6h4LM4CJDa7WBSNsWmlCWwhWn+fiLUYqVrxqjyR0imj3WQzCFndJe2wJVRGqzY\nqpvhdHalhhuA3iZ6FVrQo+OBIYyyNY2tRvHJ1SfDRhkUdmDbFot3fZmueACZhNcl\ntifWUcO8qnNRoi9hJxfO9dBOx5kwR1O3UkNSqkUdURpa9tZXpckwmhQny+gzaX24\nxFGx4RcWKiA0CSTZiD91teoPL/eSd+2NnQmebLLKvhzX2PgAzBmKlbqDzqV13THe\nWDUF0nCC0peayy6EQg1JVBZeADx6EfaPq7YpdEhbeEZUb1WhSQKCAQEA4AhEzQRL\n+ZUxkf/887dhvvM3F9hYXWBSxEgtpb4StwwGUMEwzfh0jEd2ci3OTLM3LaCfpjy8\npeLq0xH7DvKuiSqMNQ2MVHRJJ/SFzh4AgkTdfmgnXO8AoT1ig7cOgfZYd6GYSAyN\nx2cRRekzoDCtr+I2SfRvtqiL0BMr4WXtqplFfi2j3NW6muacSw8jV1OAefqK1bv9\nCCmYKkiaehS+/SZ8sCwRJQAxiap8Y9qhvnDcMEQn5ktp1EqICwL7AkZrXJjR7tEu\nN6N8l6PA7hXmDtvdBCBE27YKHFM9/nXmLuqc3OwqrXDdAnAFBWVBy4ZfnRk4f9DF\nKyCzSPqXONHH6QKCAQEAt/RtY6QFgYBc8wBPo9XAUKvc97Vhupk8s9udvralXVQn\n9xqu7BI/6bJFBeIFmPe7WF3at7s6xtkaEiRgzxVozJAaj612ocGZ4XVnWjYBwfAg\npUJm3LxlcgYjPJNPD9iW8Y5L1kkzrxGxSyg9vocmAUT82I2+kJZUjjcqXeC1qHKb\n81UMS7pwjn3Pqvypj3JGuA2QuoTQk9zGPV/snFFTwc96wATIngvQlS3gOTxRtSeQ\nNOVjooir5tDlJ0JPsFRt6Am8qwaZlQpWLCXJCS9O6lz8M+E11LkiRJ6RwGtwJyZ0\n2HyStXnm8XAIg39eSF4VdeX4oUHNOKqAvpJcqh1diwKCAQB9woaEo9XWS/bgo1em\npBmdrEAQBaVM/YZYor/lG3a8BGcDK/+uj7s1B+Sooaf+DFh8J3+XPN8VK/Vv+Nb7\nBCYP36+HFIzi18g9QpleJB9bATdo/yShmZS77o5/q+Hb+KKevTqTr2BjQCtFGGyf\njtPhONYK28leOu0JJ8npJGOu8Y0+EHa8v8mZrLytsdi/66LiVk9bPcUDsP23Nrxo\nrX6s8s7pwkL5h0tHoUlkcFV9ZzNjcM8JCupkflYyAEFRfKBGAY9PRWNxXjfRk934\niJh1XNyLtWWSBHVWgy/4yMwnKBJYPcaTSBj4hlN7POUbT9oagQ7jSRl1ZUzfObRr\n8+8JAoIBAHdbblq3gycKvcSiEAqS16zARqinFNk5MkzVE5457kQHJcNMmfDsynF8\nzRJLHCyrmPB2PUL3heSd2y+oUPQsxQnt+n39JDtCXfEO91duMWe/wb7Yvds9lmkM\nmHJmdmcmtpHXLVfFuXY/eZT1cFKeaEdsls5/8gTotcZs/t1YvyWj75JMPaUpPH0I\nHuWn+PDtSfuEOcL8CEXYevw5sFGg50Kat9+9etopelZBJPqeK7nqCMR5jlNYcIlc\nQ0Nc4E3eCk4De1pQ9vfgdBdDERUXkBG68Z1L5Fy5HwX9CT8btvKAHlKVSa/cf79q\nXOvNicRU34HXDKK9CkZZQx3j46IxWY8CggEAXWCiG1lmdpmpiR5whhqpSr5/Khbu\nmRUGcoVwh5JBPj6ABOCBaJXulMPTWm1xh87ygFA84xc0Fz1FwKaiBpf226zfvoRS\nox03yGpORuX35yxQSq9clT9E2by4ENk2qn7Vfs4QBnxBLVq6UTtYj0LtDOLC7nMC\nnPuaFzUTB/1ApkBbM6IrQthd1E1QmVh6Iha376puzRa9lOyOwU6kIjPJDIsMfeu2\nw0EN6loTNHvr+NnO+7kod7PNmUASbkMxnHx6JE+TJQCep3RUzKyDIwoAAqqa04bf\neAx7a/7e53OmTKduqmfY60DMt5/8uxGLerHAJxCa2v9oVhx0/F2yst0yWQ==\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T01:56:54.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:54 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:56:54.984 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T01:56:55.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:55 smithi016 ceph-mon[31245]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:55.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:55 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:56:57.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T01:56:57.019 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-09-21T01:56:57.894 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:57 smithi016 ceph-mon[31245]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:57.894 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:57 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:56:58.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$hMO/YdcaZY24eQvHwShLBead.v8vIuTZxsNe7t0DKJb0Rg7vnawBm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726883817, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T01:56:58.401 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-09-21T01:56:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:58 smithi016 ceph-mon[31245]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:58 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:56:59.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T01:56:59.636 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:56:59.636 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-09-21T01:56:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:56:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-09-21T01:56:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-09-21T01:56:59.645 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: Password: l7gfyrll12 2024-09-21T01:56:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:56:59.648 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config directory 2024-09-21T01:57:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:56:59 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2479175053' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T01:57:00.979 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T01:57:00.980 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T01:57:00.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.980 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:00.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:00.981 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-09-21T01:57:01.020 INFO:tasks.cephadm:Fetching config... 2024-09-21T01:57:01.021 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:01.021 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T01:57:01.037 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T01:57:01.037 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:01.037 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T01:57:01.093 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T01:57:01.093 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:01.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T01:57:01.159 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T01:57:01.160 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:01.160 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T01:57:01.215 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T01:57:01.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T01:57:01.290 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:01.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T01:57:01.412 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:01.425 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T01:57:01.492 INFO:teuthology.orchestra.run.smithi161.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc2XOqksVhGaXybPjIglgfGnyfQnqRkkAv+Pn4YLsyaZvK6g+zt2hItgwp0v/pTNLMkZ9LBuAlbnBnTUBqqOuqM4heFVkHx4K1kBm4lsAv4T+irRzIP89ldkq+cGWsOeU+rgRB7mLQ+sHhUwHtcKeSIbCYaLaRJ8rjmMIkXZW/07rCkg5C0YqTpvsdAKfZ8k0eWYz6PcjuexI5Yh72DU6XdNY2qfI9uK9KR6GijLh3Vgr/PhjxgHLzxI+2vWS4+seRmKVaRGE4JJLDC6I5cQC5NhIFfP8qis64R95daKJ8n/O0LxxNYkWNghUDqgaZ4hZ2dvPCSjTrrimlldIecQf7iT8CEUmRPPaOu3eeNLkkOQ+JXuu8vc4Gc8s4AzUE03msj3fwP2RmEKI8+V3CvG/NqMiN2X4Qi3jYwwrxjTHkC+GEAlaQOSE4C+HYMxmVN1tyw73vIzS3tP33NwBQ8VlVxfnJT3Lv5xDdtY+/tRM8DMmgAgerv1rhe703dsMZf/8= ceph-8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:01.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T01:57:01.514 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:01 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1611006324' entity='client.admin' 2024-09-21T01:57:01.761 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:03.109 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T01:57:03.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T01:57:03.597 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:03.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:03 smithi016 ceph-mon[31245]: mgrmap e13: x(active, since 12s) 2024-09-21T01:57:03.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3799743795' entity='client.admin' 2024-09-21T01:57:05.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:05.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:05.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:05.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:05.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:05.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:05.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:05 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:05.915 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-09-21T01:57:05.915 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:57:05.915 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-09-21T01:57:05.934 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:57:05.934 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:05.991 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-09-21T01:57:05.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch host add smithi123 2024-09-21T01:57:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:06.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:57:06.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:06.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:06.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:06 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:07 smithi016 ceph-mon[31245]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:08.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:08 smithi016 ceph-mon[31245]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:09 smithi016 ceph-mon[31245]: Deploying cephadm binary to smithi123 2024-09-21T01:57:10.850 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi123' with addr '172.21.15.123' 2024-09-21T01:57:11.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T01:57:11.774 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:11.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:11 smithi016 ceph-mon[31245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:11.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:11 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:11.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:11 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:11.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:11 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:11.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:11 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:12.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:12.429 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-09-21T01:57:12.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:12 smithi016 ceph-mon[31245]: Added host smithi123 2024-09-21T01:57:12.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:12 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:13.033 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi161 2024-09-21T01:57:13.033 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:57:13.033 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.conf 2024-09-21T01:57:13.052 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:57:13.052 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:13.110 INFO:tasks.cephadm:Adding host smithi161 to orchestrator... 2024-09-21T01:57:13.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch host add smithi161 2024-09-21T01:57:13.373 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:13.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:13 smithi016 ceph-mon[31245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:13.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:13 smithi016 ceph-mon[31245]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:15 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:16 smithi016 ceph-mon[31245]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:16 smithi016 ceph-mon[31245]: Deploying cephadm binary to smithi161 2024-09-21T01:57:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:16 smithi016 ceph-mon[31245]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:16 smithi016 ceph-mon[31245]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:16 smithi016 ceph-mon[31245]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:17.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:17 smithi016 ceph-mon[31245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:17.824 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi161' with addr '172.21.15.161' 2024-09-21T01:57:18.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T01:57:18.705 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: Added host smithi161 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:19.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:18 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:19.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:57:19.336 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}] 2024-09-21T01:57:19.932 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T01:57:19.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-21T01:57:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:20 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:20 smithi016 ceph-mon[31245]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:20.183 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-21T01:57:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:21 smithi016 ceph-mon[31245]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:21 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T01:57:21.762 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-09-21T01:57:21.763 INFO:tasks.cephadm:Adding mon.b on smithi123 2024-09-21T01:57:21.763 INFO:tasks.cephadm:Adding mon.c on smithi161 2024-09-21T01:57:21.763 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c' 2024-09-21T01:57:22.010 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:57:22.056 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:22 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:22.746 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mon update... 2024-09-21T01:57:23.270 DEBUG:teuthology.orchestra.run.smithi123:mon.b> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.b.service 2024-09-21T01:57:23.276 DEBUG:teuthology.orchestra.run.smithi161:mon.c> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.c.service 2024-09-21T01:57:23.279 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T01:57:23.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:57:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:23.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:23 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:24.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:24 smithi016 ceph-mon[31245]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:24.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:24 smithi016 ceph-mon[31245]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-09-21T01:57:24.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:24 smithi016 ceph-mon[31245]: Deploying daemon mon.c on smithi161 2024-09-21T01:57:25.573 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:57:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:25 smithi016 ceph-mon[31245]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:26.508 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:57:26.508 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":1,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","modified":"2024-09-21T01:55:56.591199Z","created":"2024-09-21T01:55:56.591199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:57:26.508 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 1 2024-09-21T01:57:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 podman[34041]: 2024-09-21 01:57:27.263185975 +0000 UTC m=+0.020042611 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:57:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:27 smithi016 ceph-mon[31245]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:27 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/131602264' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 podman[34041]: 2024-09-21 01:57:27.590019384 +0000 UTC m=+0.346876017 container init c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 podman[34041]: 2024-09-21 01:57:27.593186109 +0000 UTC m=+0.350042743 container start c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pidfile_write: ignore empty --pid-file 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: load: jerasure load: lrc 2024-09-21T01:57:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: RocksDB version: 7.9.2 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Git sha 0 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: DB SUMMARY 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: DB Session ID: DDV3NP6GRPFDPWSC79FP 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: CURRENT file: CURRENT 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-21T01:57:27.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.error_if_exists: 0 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.create_if_missing: 0 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.paranoid_checks: 1 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.env: 0x55949575fd60 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.info_log: 0x55949a9d7360 2024-09-21T01:57:27.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T01:57:27.919 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.statistics: (nil) 2024-09-21T01:57:27.919 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.use_fsync: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_log_file_size: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_fallocate: 1 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.use_direct_reads: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T01:57:27.920 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.db_log_dir: 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.wal_dir: 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.write_buffer_manager: 0x55949a9dbd60 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T01:57:27.921 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.unordered_write: 0 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T01:57:27.922 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.row_cache: None 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.wal_filter: None 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.two_write_queues: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.wal_compression: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.atomic_flush: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T01:57:27.923 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.log_readahead_size: 0 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_background_jobs: 2 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_background_compactions: -1 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_subcompactions: 1 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T01:57:27.924 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_open_files: -1 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T01:57:27.925 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_background_flushes: -1 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Compression algorithms supported: 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kZSTD supported: 0 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kXpressCompression supported: 0 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kBZip2Compression supported: 0 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kLZ4Compression supported: 1 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kZlibCompression supported: 1 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: kSnappyCompression supported: 1 2024-09-21T01:57:27.926 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.merge_operator: 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_filter: None 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55949a9d6fa0) 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout: cache_index_and_filter_blocks: 1 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T01:57:27.927 INFO:journalctl@ceph.mon.c.smithi161.stdout: pin_top_level_index_and_filter: 1 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: index_type: 0 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: data_block_index_type: 0 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: index_shortening: 1 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: checksum: 4 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: no_block_cache: 0 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_cache: 0x55949699b610 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_cache_name: BinnedLRUCache 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_cache_options: 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: capacity : 536870912 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: num_shard_bits : 4 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: strict_capacity_limit : 0 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: high_pri_pool_ratio: 0.000 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_cache_compressed: (nil) 2024-09-21T01:57:27.928 INFO:journalctl@ceph.mon.c.smithi161.stdout: persistent_cache: (nil) 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_size: 4096 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_size_deviation: 10 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_restart_interval: 16 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: index_block_restart_interval: 1 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: metadata_block_size: 4096 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: partition_filters: 0 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: use_delta_encoding: 1 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: filter_policy: bloomfilter 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: whole_key_filtering: 1 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: verify_compression: 0 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: read_amp_bytes_per_bit: 0 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: format_version: 5 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: enable_index_compression: 1 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: block_align: 0 2024-09-21T01:57:27.929 INFO:journalctl@ceph.mon.c.smithi161.stdout: max_auto_readahead_size: 262144 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout: prepopulate_block_cache: 0 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout: initial_auto_readahead_size: 8192 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression: NoCompression 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.num_levels: 7 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T01:57:27.930 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:57:27.931 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T01:57:27.932 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T01:57:27.933 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.inplace_update_support: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.bloom_locality: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.max_successive_merges: 0 2024-09-21T01:57:27.934 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.ttl: 2592000 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enable_blob_files: false 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.min_blob_size: 0 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T01:57:27.935 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 56d569e8-6e3c-42a7-a5ad-cb0157f590db 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883847629722, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883847630158, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726883847, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "56d569e8-6e3c-42a7-a5ad-cb0157f590db", "db_session_id": "DDV3NP6GRPFDPWSC79FP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883847630254, "job": 1, "event": "recovery_finished"} 2024-09-21T01:57:27.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55949a9faa80 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: DB pointer 0x55949b7a0000 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** DB Stats ** 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.937 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** Compaction Stats [default] ** 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** Compaction Stats [default] ** 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.938 INFO:journalctl@ceph.mon.c.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Block cache BinnedLRUCache@0x55949699b610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.939 INFO:journalctl@ceph.mon.c.smithi161.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: using public_addr v2:172.21.15.161:0/0 -> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: starting mon.c rank -1 at public addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] at bind addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(???) e0 preinit fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: e1 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: btime 2024-09-21T01:56:00:153260+0000 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: legacy client fscid: -1 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout: No filesystems configured 2024-09-21T01:57:27.940 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Added host smithi123 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Deploying cephadm binary to smithi161 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Added host smithi161 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:27.944 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:27.945 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: Deploying daemon mon.c on smithi161 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/131602264' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 ceph-mon[34061]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 bash[34041]: c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 2024-09-21T01:57:27.946 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:27 smithi161 systemd[1]: Started Ceph mon.c for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:57:28.165 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T01:57:28.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:57:28.413 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: mon.a calling monitor election 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: mon.c calling monitor election 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: monmap epoch 2 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: last_changed 2024-09-21T01:57:27.658415+0000 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: min_mon_release 19 (squid) 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: election_strategy: 1 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: fsmap 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: mgrmap e13: x(active, since 42s) 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: overall HEALTH_OK 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:32 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:33.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:33.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: mon.a calling monitor election 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: mon.c calling monitor election 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: monmap epoch 2 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: last_changed 2024-09-21T01:57:27.658415+0000 2024-09-21T01:57:33.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: min_mon_release 19 (squid) 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: election_strategy: 1 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: fsmap 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: mgrmap e13: x(active, since 42s) 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: overall HEALTH_OK 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:33.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:32 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:33.571 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:57:33.572 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","modified":"2024-09-21T01:57:27.658415Z","created":"2024-09-21T01:55:56.591199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T01:57:33.572 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 2 2024-09-21T01:57:33.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:33 smithi161 ceph-mon[34061]: Deploying daemon mon.b on smithi123 2024-09-21T01:57:33.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:33 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/2664673157' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:33.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:33 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:33 smithi016 ceph-mon[31245]: Deploying daemon mon.b on smithi123 2024-09-21T01:57:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:33 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/2664673157' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:33 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:34 smithi016 ceph-mon[31245]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:35.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:34 smithi161 ceph-mon[34061]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:35.318 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T01:57:35.319 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:57:35.593 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:57:36.211 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 podman[33434]: 2024-09-21 01:57:36.210394481 +0000 UTC m=+0.120134169 container create 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T01:57:36.299 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:57:36.300 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","modified":"2024-09-21T01:57:27.658415Z","created":"2024-09-21T01:55:56.591199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T01:57:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 2 2024-09-21T01:57:36.472 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 podman[33434]: 2024-09-21 01:57:36.437746908 +0000 UTC m=+0.347486605 container init 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T01:57:36.472 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 podman[33434]: 2024-09-21 01:57:36.44094286 +0000 UTC m=+0.350682549 container start 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T01:57:36.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pidfile_write: ignore empty --pid-file 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: load: jerasure load: lrc 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: RocksDB version: 7.9.2 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Git sha 0 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: DB SUMMARY 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: DB Session ID: OB25J9U0WZ5PZRG0T0BQ 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: CURRENT file: CURRENT 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T01:57:36.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.error_if_exists: 0 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.create_if_missing: 0 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.paranoid_checks: 1 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.env: 0x55e3d90e2d60 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.info_log: 0x55e3debd5360 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.statistics: (nil) 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.use_fsync: 0 2024-09-21T01:57:36.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_log_file_size: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_fallocate: 1 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.use_direct_reads: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.db_log_dir: 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.wal_dir: 2024-09-21T01:57:36.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T01:57:36.775 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.write_buffer_manager: 0x55e3debd9d60 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.unordered_write: 0 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T01:57:36.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.row_cache: None 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.wal_filter: None 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.two_write_queues: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.wal_compression: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.atomic_flush: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.log_readahead_size: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T01:57:36.777 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_background_jobs: 2 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_background_compactions: -1 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_subcompactions: 1 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T01:57:36.778 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_open_files: -1 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_background_flushes: -1 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Compression algorithms supported: 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kZSTD supported: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kXpressCompression supported: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kBZip2Compression supported: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kLZ4Compression supported: 1 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kZlibCompression supported: 1 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T01:57:36.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: kSnappyCompression supported: 1 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.merge_operator: 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_filter: None 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3debd4fa0) 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout: cache_index_and_filter_blocks: 1 2024-09-21T01:57:36.780 INFO:journalctl@ceph.mon.b.smithi123.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: pin_top_level_index_and_filter: 1 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: index_type: 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: data_block_index_type: 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: index_shortening: 1 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: checksum: 4 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: no_block_cache: 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_cache: 0x55e3dab99610 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_cache_name: BinnedLRUCache 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_cache_options: 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: capacity : 536870912 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: num_shard_bits : 4 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: strict_capacity_limit : 0 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: high_pri_pool_ratio: 0.000 2024-09-21T01:57:36.781 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_cache_compressed: (nil) 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: persistent_cache: (nil) 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_size: 4096 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_size_deviation: 10 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_restart_interval: 16 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: index_block_restart_interval: 1 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: metadata_block_size: 4096 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: partition_filters: 0 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: use_delta_encoding: 1 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: filter_policy: bloomfilter 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: whole_key_filtering: 1 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: verify_compression: 0 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: read_amp_bytes_per_bit: 0 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: format_version: 5 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: enable_index_compression: 1 2024-09-21T01:57:36.782 INFO:journalctl@ceph.mon.b.smithi123.stdout: block_align: 0 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout: max_auto_readahead_size: 262144 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout: prepopulate_block_cache: 0 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout: initial_auto_readahead_size: 8192 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression: NoCompression 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.num_levels: 7 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T01:57:36.783 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:57:36.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T01:57:36.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T01:57:36.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.inplace_update_support: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.bloom_locality: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.max_successive_merges: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T01:57:36.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.ttl: 2592000 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enable_blob_files: false 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.min_blob_size: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T01:57:36.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0507d227-1536-414c-b9aa-71fbb987e828 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883856477438, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883856477852, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726883856, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0507d227-1536-414c-b9aa-71fbb987e828", "db_session_id": "OB25J9U0WZ5PZRG0T0BQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726883856477943, "job": 1, "event": "recovery_finished"} 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3debf8a80 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: DB pointer 0x55e3df99e000 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T01:57:36.789 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** DB Stats ** 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** Compaction Stats [default] ** 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:36.790 INFO:journalctl@ceph.mon.b.smithi123.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** Compaction Stats [default] ** 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: Block cache BinnedLRUCache@0x55e3dab99610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: using public_addr v2:172.21.15.123:0/0 -> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: starting mon.b rank -1 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(???) e0 preinit fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T01:57:36.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: e1 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: btime 2024-09-21T01:56:00:153260+0000 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: legacy client fscid: -1 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout: No filesystems configured 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T01:57:36.793 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Added host smithi123 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Deploying cephadm binary to smithi161 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.795 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Added host smithi161 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2940241059' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.796 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:36.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Deploying daemon mon.c on smithi161 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/131602264' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.a calling monitor election 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.c calling monitor election 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: monmap epoch 2 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: last_changed 2024-09-21T01:57:27.658415+0000 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: min_mon_release 19 (squid) 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: election_strategy: 1 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: fsmap 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mgrmap e13: x(active, since 42s) 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: overall HEALTH_OK 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: Deploying daemon mon.b on smithi123 2024-09-21T01:57:36.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/2664673157' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:36.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:36.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:36.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 ceph-mon[33453]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T01:57:36.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 bash[33434]: 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 2024-09-21T01:57:36.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:36 smithi123 systemd[1]: Started Ceph mon.b for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:57:37.897 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T01:57:37.898 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:57:38.149 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:57:41.902 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:57:41.903 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":3,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","modified":"2024-09-21T01:57:36.504789Z","created":"2024-09-21T01:55:56.591199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T01:57:41.903 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 3 2024-09-21T01:57:41.903 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.903 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:41.903 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.903 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: mon.a calling monitor election 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: mon.c calling monitor election 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: mon.b calling monitor election 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: monmap epoch 3 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: last_changed 2024-09-21T01:57:36.504789+0000 2024-09-21T01:57:41.904 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: min_mon_release 19 (squid) 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: election_strategy: 1 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: fsmap 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: mgrmap e13: x(active, since 51s) 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: overall HEALTH_OK 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:41 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: mon.a calling monitor election 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: mon.c calling monitor election 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: mon.b calling monitor election 2024-09-21T01:57:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: monmap epoch 3 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: last_changed 2024-09-21T01:57:36.504789+0000 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: min_mon_release 19 (squid) 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: election_strategy: 1 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: fsmap 2024-09-21T01:57:41.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: mgrmap e13: x(active, since 51s) 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: overall HEALTH_OK 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:41.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:41 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: mon.a calling monitor election 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: mon.c calling monitor election 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: mon.b calling monitor election 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: monmap epoch 3 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: last_changed 2024-09-21T01:57:36.504789+0000 2024-09-21T01:57:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: created 2024-09-21T01:55:56.591199+0000 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: min_mon_release 19 (squid) 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: election_strategy: 1 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.c 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: 2: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: fsmap 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: mgrmap e13: x(active, since 51s) 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: overall HEALTH_OK 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:42.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:41 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:42.559 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T01:57:42.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T01:57:42.815 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:42 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:42.815 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:42 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/1224524061' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:42.815 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:42 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:42 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:42 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/1224524061' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:42 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:42 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:42 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/1224524061' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:57:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:42 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:43.449 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:43.449 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-21T01:57:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T01:57:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-09-21T01:57:43.895 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2844537705' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.896 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.897 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:43 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2844537705' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:43 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:57:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2844537705' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:43.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:43 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.186 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T01:57:44.186 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:44.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T01:57:44.216 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:57:44.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:44.282 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:57:44.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T01:57:44.311 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:57:44.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:44.377 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:57:44.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T01:57:44.406 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:57:44.406 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:57:44.472 INFO:tasks.cephadm:enabling module snap_schedule 2024-09-21T01:57:44.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:44.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:44 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:44.903 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T01:57:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:44 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:57:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:57:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:45.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:44 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:45.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: Reconfiguring daemon mon.a on smithi016 2024-09-21T01:57:45.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:45.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:45 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: Reconfiguring daemon mon.a on smithi016 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:45.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:45.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:45 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: Reconfiguring daemon mon.a on smithi016 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:45.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:45 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setuser ceph since I am not root 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ignoring --setgroup ceph since I am not root 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: Reconfiguring mon.b (monmap changed)... 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: Reconfiguring daemon mon.b on smithi123 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:46.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:46.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:46.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:46.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:46 smithi016 ceph-mon[31245]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:46.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: Reconfiguring mon.b (monmap changed)... 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: Reconfiguring daemon mon.b on smithi123 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:46.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:46 smithi161 ceph-mon[34061]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:47.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: Reconfiguring mon.b (monmap changed)... 2024-09-21T01:57:47.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: Reconfiguring daemon mon.b on smithi123 2024-09-21T01:57:47.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "snap_schedule"} : dispatch 2024-09-21T01:57:47.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:47.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' 2024-09-21T01:57:47.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:57:47.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:57:47.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:46 smithi123 ceph-mon[33453]: from='mgr.14152 172.21.15.16:0/498389168' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:57:47.284 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-09-21T01:57:47.284 INFO:tasks.cephadm:Adding mgr.y on smithi123 2024-09-21T01:57:47.284 INFO:tasks.cephadm:Adding mgr.z on smithi161 2024-09-21T01:57:47.284 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch apply mgr '3;smithi016=x;smithi123=y;smithi161=z' 2024-09-21T01:57:47.859 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:57:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:47 smithi016 ceph-mon[31245]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:47 smithi016 ceph-mon[31245]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:57:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:47 smithi016 ceph-mon[31245]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:57:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:47 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T01:57:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:47 smithi016 ceph-mon[31245]: mgrmap e14: x(active, since 56s) 2024-09-21T01:57:47.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:47 smithi161 ceph-mon[34061]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:47 smithi161 ceph-mon[34061]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:57:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:47 smithi161 ceph-mon[34061]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:57:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:47 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T01:57:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:47 smithi161 ceph-mon[34061]: mgrmap e14: x(active, since 56s) 2024-09-21T01:57:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:47 smithi123 ceph-mon[33453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:57:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:47 smithi123 ceph-mon[33453]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:57:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:47 smithi123 ceph-mon[33453]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:57:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:47 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2193822207' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-21T01:57:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:47 smithi123 ceph-mon[33453]: mgrmap e14: x(active, since 56s) 2024-09-21T01:57:52.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: Active manager daemon x restarted 2024-09-21T01:57:52.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: Activating manager daemon x 2024-09-21T01:57:52.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:57:52.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: mgrmap e15: x(active, starting, since 0.00474799s) 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:57:52.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:51 smithi123 ceph-mon[33453]: Manager daemon x is now available 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: Active manager daemon x restarted 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: Activating manager daemon x 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: mgrmap e15: x(active, starting, since 0.00474799s) 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:52.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:57:52.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:57:52.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:57:52.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:57:52.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:51 smithi016 ceph-mon[31245]: Manager daemon x is now available 2024-09-21T01:57:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: Active manager daemon x restarted 2024-09-21T01:57:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: Activating manager daemon x 2024-09-21T01:57:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:57:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: mgrmap e15: x(active, starting, since 0.00474799s) 2024-09-21T01:57:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:57:52.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:51 smithi161 ceph-mon[34061]: Manager daemon x is now available 2024-09-21T01:57:52.943 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mgr update... 2024-09-21T01:57:53.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:53.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:57:53.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:57:53.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:52 smithi016 ceph-mon[31245]: mgrmap e16: x(active, since 1.01116s) 2024-09-21T01:57:53.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:53.246 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:53.246 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:57:53.246 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:57:53.246 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:52 smithi161 ceph-mon[34061]: mgrmap e16: x(active, since 1.01116s) 2024-09-21T01:57:53.246 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:57:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:57:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-21T01:57:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:52 smithi123 ceph-mon[33453]: mgrmap e16: x(active, since 1.01116s) 2024-09-21T01:57:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:53.507 DEBUG:teuthology.orchestra.run.smithi123:mgr.y> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.y.service 2024-09-21T01:57:53.509 DEBUG:teuthology.orchestra.run.smithi161:mgr.z> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.z.service 2024-09-21T01:57:53.512 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-21T01:57:53.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-21T01:57:53.514 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-21T01:57:53.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-21T01:57:53.556 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-21T01:57:53.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-21T01:57:53.599 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-21T01:57:53.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-21T01:57:53.642 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-21T01:57:53.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-21T01:57:53.686 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-21T01:57:53.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-21T01:57:53.729 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-21T01:57:53.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-21T01:57:53.772 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-21T01:57:53.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:57:53.692+0000 7f5951f2d640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:01:57:53] ENGINE Error in HTTPServer.serve 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: Traceback (most recent call last): 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._connections.run(self.expiration_interval) 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:57:53.813 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._run(expiration_interval) 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: s = self.context.wrap_socket( 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: return self.sslsocket_class._create( 2024-09-21T01:57:53.814 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:57:53.815 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self.do_handshake() 2024-09-21T01:57:53.815 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:57:53.815 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: self._sslobj.do_handshake() 2024-09-21T01:57:53.815 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:57:53.815 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 01:57:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T01:57:53.817 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-21T01:57:53.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-09-21T01:57:53.843 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:53.845 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-09-21T01:57:53.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_merge_size 5 2024-09-21T01:57:53.891 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-09-21T01:57:53.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_bits 3 2024-09-21T01:57:53.935 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-09-21T01:57:53.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_bal_split_size 100 2024-09-21T01:57:54.000 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-21T01:57:54.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-21T01:57:54.044 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-21T01:57:54.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-21T01:57:54.091 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-21T01:57:54.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-21T01:57:54.152 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-21T01:57:54.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-21T01:57:54.201 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-21T01:57:54.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-21T01:57:54.257 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T01:57:54.258 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-21T01:57:54.306 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T01:57:54.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-21T01:57:54.352 INFO:tasks.cephadm:setting config [mgr] client_mount_timeout = 30 2024-09-21T01:57:54.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr client_mount_timeout 30 2024-09-21T01:57:54.402 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-09-21T01:57:54.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr debug_cephsqlite 20 2024-09-21T01:57:54.450 INFO:tasks.cephadm:setting config [mgr] debug_client = 20 2024-09-21T01:57:54.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr debug_client 20 2024-09-21T01:57:54.514 INFO:tasks.cephadm:setting config [mgr] debug_mgr = 20 2024-09-21T01:57:54.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr debug_mgr 20 2024-09-21T01:57:54.576 INFO:tasks.cephadm:setting config [mgr] debug_ms = 1 2024-09-21T01:57:54.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr debug_ms 1 2024-09-21T01:57:54.619 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-09-21T01:57:54.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-09-21T01:57:54.675 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-09-21T01:57:54.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-09-21T01:57:54.724 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-09-21T01:57:54.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-09-21T01:57:54.776 INFO:tasks.cephadm:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T01:57:54.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T01:57:54.852 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-21T01:57:54.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: Saving service mgr spec with placement smithi016=x;smithi123=y;smithi161=z;count:3 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: [21/Sep/2024:01:57:53] ENGINE Bus STARTING 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: [21/Sep/2024:01:57:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-21T01:57:54.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: [21/Sep/2024:01:57:53] ENGINE Error in HTTPServer.serve 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._run(expiration_interval) 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: s = self.context.wrap_socket( 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.sslsocket_class._create( 2024-09-21T01:57:54.877 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.do_handshake() 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: self._sslobj.do_handshake() 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:54.883 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-21T01:57:54.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: Saving service mgr spec with placement smithi016=x;smithi123=y;smithi161=z;count:3 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: [21/Sep/2024:01:57:53] ENGINE Bus STARTING 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: [21/Sep/2024:01:57:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: [21/Sep/2024:01:57:53] ENGINE Error in HTTPServer.serve 2024-09-21T01:57:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout: Traceback (most recent call last): 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._run(expiration_interval) 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: s = self.context.wrap_socket( 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: return self.sslsocket_class._create( 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: self.do_handshake() 2024-09-21T01:57:54.916 INFO:journalctl@ceph.mon.c.smithi161.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout: self._sslobj.do_handshake() 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:54.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:54.951 DEBUG:tasks.cephadm:set 30 configs 2024-09-21T01:57:54.951 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi016') 2024-09-21T01:57:54.974 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: Saving service mgr spec with placement smithi016=x;smithi123=y;smithi161=z;count:3 2024-09-21T01:57:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: [21/Sep/2024:01:57:53] ENGINE Bus STARTING 2024-09-21T01:57:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: [21/Sep/2024:01:57:53] ENGINE Serving on https://172.21.15.16:7150 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: [21/Sep/2024:01:57:53] ENGINE Error in HTTPServer.serve 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: Traceback (most recent call last): 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._run(expiration_interval) 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: s = self.context.wrap_socket( 2024-09-21T01:57:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: return self.sslsocket_class._create( 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: self.do_handshake() 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: self._sslobj.do_handshake() 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.893 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.902 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: [21/Sep/2024:01:57:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-21T01:57:55.902 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: [21/Sep/2024:01:57:53] ENGINE Bus STARTED 2024-09-21T01:57:55.902 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.902 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:55.903 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:55.903 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: mgrmap e17: x(active, since 2s) 2024-09-21T01:57:55.903 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:55 smithi016 ceph-mon[31245]: from='client.? ' entity='client.admin' 2024-09-21T01:57:55.904 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:57:55.930 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.931 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.932 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.939 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: [21/Sep/2024:01:57:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: [21/Sep/2024:01:57:53] ENGINE Bus STARTED 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: mgrmap e17: x(active, since 2s) 2024-09-21T01:57:56.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:55 smithi123 ceph-mon[33453]: from='client.? ' entity='client.admin' 2024-09-21T01:57:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: [21/Sep/2024:01:57:53] ENGINE Serving on http://172.21.15.16:8765 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: [21/Sep/2024:01:57:53] ENGINE Bus STARTED 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"} : dispatch 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: mgrmap e17: x(active, since 2s) 2024-09-21T01:57:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:55 smithi161 ceph-mon[34061]: from='client.? ' entity='client.admin' 2024-09-21T01:57:56.693 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:56.694 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:57:58.040 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:57 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2732288031' entity='client.admin' 2024-09-21T01:57:58.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:57 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2732288031' entity='client.admin' 2024-09-21T01:57:58.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:57 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2732288031' entity='client.admin' 2024-09-21T01:57:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e9c03fb70 msgr2=0x7f2e9c042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e9c03fb70 0x7f2e9c042080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4005d90 tx=0x7f2ea4004d30 comp rx=0 tx=0).stop 2024-09-21T01:57:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb41083b0 msgr2=0x7f2eb419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb41083b0 0x7f2eb419ad70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00e970 tx=0x7f2eac00ed70 comp rx=0 tx=0).stop 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 shutdown_connections 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e9c03fb70 0x7f2e9c042080 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb41083b0 0x7f2eb419ad70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2eb4105790 0x7f2eb419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 --2- 172.21.15.16:0/4145354962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb4102d30 0x7f2eb4199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 >> 172.21.15.16:0/4145354962 conn(0x7f2eb40fd770 msgr2=0x7f2eb40fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 shutdown_connections 2024-09-21T01:57:58.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.421+0000 7f2ebb914640 1 -- 172.21.15.16:0/4145354962 wait complete. 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 -- 172.21.15.16:0/1331713710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e8105980 msgr2=0x7fa7e8105df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 --2- 172.21.15.16:0/1331713710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e8105980 0x7fa7e8105df0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009900 tx=0x7fa7e402a480 comp rx=0 tx=0).stop 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 -- 172.21.15.16:0/1331713710 shutdown_connections 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 --2- 172.21.15.16:0/1331713710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e81073b0 0x7fa7e810dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 --2- 172.21.15.16:0/1331713710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e8106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 --2- 172.21.15.16:0/1331713710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e8105980 0x7fa7e8105df0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 -- 172.21.15.16:0/1331713710 >> 172.21.15.16:0/1331713710 conn(0x7fa7e81012f0 msgr2=0x7fa7e81037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 -- 172.21.15.16:0/1331713710 shutdown_connections 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.425+0000 7fa7efc54640 1 -- 172.21.15.16:0/1331713710 wait complete. 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 Processor -- start 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 -- start start 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 0x7fa7e819beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e8111170 con 0x7fa7e81073b0 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e8110ff0 con 0x7fa7e8105980 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7efc54640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e81112f0 con 0x7fa7e8106a10 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:58.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36016/0 (socket says 172.21.15.16:36016) 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 -- 172.21.15.16:0/2920663285 learned_addr learned my addr 172.21.15.16:0/2920663285 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ed9f0640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 0x7fa7e819beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ed1ef640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 -- 172.21.15.16:0/2920663285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 msgr2=0x7fa7e819c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e819c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 -- 172.21.15.16:0/2920663285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 msgr2=0x7fa7e819beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 0x7fa7e819beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.426+0000 7fa7ee1f1640 1 -- 172.21.15.16:0/2920663285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e819d850 con 0x7fa7e81073b0 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7ed1ef640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e819c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7ed9f0640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 0x7fa7e819beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7ee1f1640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000cce0 tx=0x7fa7e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:57:58.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e000f450 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e819db40 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(22 keys) ==== 976+0+0 (secure 0 0 0) 0x7fa7e000fa20 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e00188a0 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.427+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e81a42d0 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.428+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e8105df0 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.428+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fa7e00213f0 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.428+0000 7fa7deffd640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 0x7fa7d4042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.429+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa7e00548e0 con 0x7fa7e81073b0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.429+0000 7fa7ed9f0640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 0x7fa7d4042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.429+0000 7fa7ed9f0640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 0x7fa7d4042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005d90 tx=0x7fa7e4005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:57:58.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.434+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa7e0014030 con 0x7fa7e81073b0 2024-09-21T01:57:58.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.626+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fa7e8063fc0 con 0x7fa7e81073b0 2024-09-21T01:57:58.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.629+0000 7fa7deffd640 1 -- 172.21.15.16:0/2920663285 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v13)=0 v13) ==== 117+0+0 (secure 0 0 0) 0x7fa7e0014210 con 0x7fa7e81073b0 2024-09-21T01:57:58.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 msgr2=0x7fa7d4042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 0x7fa7d4042080 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005d90 tx=0x7fa7e4005a20 comp rx=0 tx=0).stop 2024-09-21T01:57:58.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 msgr2=0x7fa7e819d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000cce0 tx=0x7fa7e000a620 comp rx=0 tx=0).stop 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 shutdown_connections 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa7d403fb70 0x7fa7d4042080 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7e81073b0 0x7fa7e819d0c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7e8106a10 0x7fa7e819c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 --2- 172.21.15.16:0/2920663285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7e8105980 0x7fa7e819beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 >> 172.21.15.16:0/2920663285 conn(0x7fa7e81012f0 msgr2=0x7fa7e8109d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.631+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 shutdown_connections 2024-09-21T01:57:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:58.632+0000 7fa7efc54640 1 -- 172.21.15.16:0/2920663285 wait complete. 2024-09-21T01:57:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 -- 172.21.15.16:0/2589191072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb264104070 msgr2=0x7fb26410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 --2- 172.21.15.16:0/2589191072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb264104070 0x7fb26410aa10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009900 tx=0x7fb25c02a480 comp rx=0 tx=0).stop 2024-09-21T01:57:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 -- 172.21.15.16:0/2589191072 shutdown_connections 2024-09-21T01:57:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 --2- 172.21.15.16:0/2589191072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb264104070 0x7fb26410aa10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 --2- 172.21.15.16:0/2589191072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2641036d0 0x7fb264103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 --2- 172.21.15.16:0/2589191072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 0x7fb264102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 -- 172.21.15.16:0/2589191072 >> 172.21.15.16:0/2589191072 conn(0x7fb2640fdfb0 msgr2=0x7fb264100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 -- 172.21.15.16:0/2589191072 shutdown_connections 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.474+0000 7fb26c075640 1 -- 172.21.15.16:0/2589191072 wait complete. 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 Processor -- start 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 -- start start 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 0x7fb264199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb264104070 0x7fb26419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb26410de40 con 0x7fb2641036d0 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb26410dcc0 con 0x7fb264102640 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26c075640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb26410dfc0 con 0x7fb264104070 2024-09-21T01:57:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36056/0 (socket says 172.21.15.16:36056) 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 -- 172.21.15.16:0/1967626546 learned_addr learned my addr 172.21.15.16:0/1967626546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269e11640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 0x7fb264199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb26a612640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb264104070 0x7fb26419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 -- 172.21.15.16:0/1967626546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb264104070 msgr2=0x7fb26419ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb264104070 0x7fb26419ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 -- 172.21.15.16:0/1967626546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 msgr2=0x7fb264199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 0x7fb264199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.475+0000 7fb269610640 1 -- 172.21.15.16:0/1967626546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26419b640 con 0x7fb2641036d0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb269610640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb25800e9f0 tx=0x7fb25800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25800cd80 con 0x7fb2641036d0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(22 keys) ==== 976+0+0 (secure 0 0 0) 0x7fb258005450 con 0x7fb2641036d0 2024-09-21T01:57:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258004db0 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb26419b930 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.476+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2641a2030 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.477+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fb258005020 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.477+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26410e600 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.477+0000 7fb262ffd640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 0x7fb24c042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.477+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb258054a70 con 0x7fb2641036d0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.478+0000 7fb269e11640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 0x7fb24c042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:57:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.478+0000 7fb269e11640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 0x7fb24c042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb254005fd0 tx=0x7fb254009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:57:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.483+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb258016330 con 0x7fb2641036d0 2024-09-21T01:57:59.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.596+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7fb258056e50 con 0x7fb2641036d0 2024-09-21T01:57:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:59 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4145354962' entity='client.admin' 2024-09-21T01:57:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:57:59 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2920663285' entity='client.admin' 2024-09-21T01:57:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.681+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7fb2641a2350 con 0x7fb2641036d0 2024-09-21T01:57:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.684+0000 7fb262ffd640 1 -- 172.21.15.16:0/1967626546 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v15)=0 v15) ==== 127+0+0 (secure 0 0 0) 0x7fb258005600 con 0x7fb2641036d0 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 msgr2=0x7fb24c042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 0x7fb24c042080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb254005fd0 tx=0x7fb254009500 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 msgr2=0x7fb26419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb25800e9f0 tx=0x7fb25800edf0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 shutdown_connections 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb24c03fb70 0x7fb24c042080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb264104070 0x7fb26419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2641036d0 0x7fb26419a280 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 --2- 172.21.15.16:0/1967626546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb264102640 0x7fb264199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 >> 172.21.15.16:0/1967626546 conn(0x7fb2640fdfb0 msgr2=0x7fb2640ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 shutdown_connections 2024-09-21T01:57:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.687+0000 7fb26c075640 1 -- 172.21.15.16:0/1967626546 wait complete. 2024-09-21T01:57:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:59 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4145354962' entity='client.admin' 2024-09-21T01:57:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:57:59 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2920663285' entity='client.admin' 2024-09-21T01:57:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:59 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4145354962' entity='client.admin' 2024-09-21T01:57:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:57:59 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2920663285' entity='client.admin' 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 -- 172.21.15.16:0/3488809392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6228103750 msgr2=0x7f6228103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 --2- 172.21.15.16:0/3488809392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6228103750 0x7f6228103bc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6224009900 tx=0x7f622402a480 comp rx=0 tx=0).stop 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 -- 172.21.15.16:0/3488809392 shutdown_connections 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 --2- 172.21.15.16:0/3488809392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228105180 0x7f622810bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 --2- 172.21.15.16:0/3488809392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62281047e0 0x7f6228104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 --2- 172.21.15.16:0/3488809392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6228103750 0x7f6228103bc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 -- 172.21.15.16:0/3488809392 >> 172.21.15.16:0/3488809392 conn(0x7f62280ff0e0 msgr2=0x7f6228101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 -- 172.21.15.16:0/3488809392 shutdown_connections 2024-09-21T01:58:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 -- 172.21.15.16:0/3488809392 wait complete. 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.176+0000 7f622f7b3640 1 Processor -- start 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 -- start start 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 0x7f6228076e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6228105180 0x7f6228075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f622810f070 con 0x7f62281047e0 2024-09-21T01:58:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f622810eef0 con 0x7f6228105180 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622f7b3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f622810f1f0 con 0x7f6228103750 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36076/0 (socket says 172.21.15.16:36076) 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 -- 172.21.15.16:0/350393648 learned_addr learned my addr 172.21.15.16:0/350393648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622dd50640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6228105180 0x7f6228075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622d54f640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 0x7f6228076e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 -- 172.21.15.16:0/350393648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 msgr2=0x7f6228076e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 0x7f6228076e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 -- 172.21.15.16:0/350393648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6228105180 msgr2=0x7f6228075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6228105180 0x7f6228075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 -- 172.21.15.16:0/350393648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6228076350 con 0x7f62281047e0 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622d54f640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 0x7f6228076e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:58:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.177+0000 7f622cd4e640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f621800d950 tx=0x7f621800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.178+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6218017030 con 0x7f62281047e0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.178+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f6218004020 con 0x7f62281047e0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.178+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62281a3e50 con 0x7f62281047e0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.178+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6218005470 con 0x7f62281047e0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.178+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62281a4240 con 0x7f62281047e0 2024-09-21T01:58:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.179+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f622810f830 con 0x7f62281047e0 2024-09-21T01:58:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.179+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f62180047a0 con 0x7f62281047e0 2024-09-21T01:58:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.179+0000 7f621e7fc640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 0x7f6214042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.179+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6218053fb0 con 0x7f62281047e0 2024-09-21T01:58:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.179+0000 7f622d54f640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 0x7f6214042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.180+0000 7f622d54f640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 0x7f6214042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f622402a9a0 tx=0x7f6224005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.184+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f621800fce0 con 0x7f62281047e0 2024-09-21T01:58:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 -- 172.21.15.16:0/649034664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5806c340 msgr2=0x7fcf581063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/649034664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5806c340 0x7fcf581063c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009900 tx=0x7fcf4802a410 comp rx=0 tx=0).stop 2024-09-21T01:58:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 -- 172.21.15.16:0/649034664 shutdown_connections 2024-09-21T01:58:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/649034664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5810b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/649034664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf58106900 0x7fcf58108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/649034664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf5806c340 0x7fcf581063c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 -- 172.21.15.16:0/649034664 >> 172.21.15.16:0/649034664 conn(0x7fcf580fdfe0 msgr2=0x7fcf58100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:00.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 -- 172.21.15.16:0/649034664 shutdown_connections 2024-09-21T01:58:00.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.355+0000 7fcf5e43c640 1 -- 172.21.15.16:0/649034664 wait complete. 2024-09-21T01:58:00.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 Processor -- start 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 -- start start 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf5806c340 0x7fcf58199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5819af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5810efe0 con 0x7fcf58106900 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5810ee60 con 0x7fcf5806c340 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.356+0000 7fcf5e43c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5810f160 con 0x7fcf581094a0 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36092/0 (socket says 172.21.15.16:36092) 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 -- 172.21.15.16:0/1780243820 learned_addr learned my addr 172.21.15.16:0/1780243820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf57fff640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf5806c340 0x7fcf58199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf5c9d9640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5819af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 -- 172.21.15.16:0/1780243820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 msgr2=0x7fcf5819af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5819af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 -- 172.21.15.16:0/1780243820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf5806c340 msgr2=0x7fcf58199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf5806c340 0x7fcf58199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 -- 172.21.15.16:0/1780243820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5819b780 con 0x7fcf58106900 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf5c9d9640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5819af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf577fe640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00d950 tx=0x7fcf4c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c017070 con 0x7fcf58106900 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7fcf4c009c30 con 0x7fcf58106900 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.357+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5819ba70 con 0x7fcf58106900 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.358+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c00f3c0 con 0x7fcf58106900 2024-09-21T01:58:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.358+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf581a2170 con 0x7fcf58106900 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.359+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5810f7a0 con 0x7fcf58106900 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.359+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fcf4c00f560 con 0x7fcf58106900 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.359+0000 7fcf557fa640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 0x7fcf40042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.359+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcf4c054260 con 0x7fcf58106900 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.359+0000 7fcf57fff640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 0x7fcf40042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.360+0000 7fcf57fff640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 0x7fcf40042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf48000c00 tx=0x7fcf48009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.364+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcf4c01e120 con 0x7fcf58106900 2024-09-21T01:58:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.377+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f622802e890 con 0x7f62281047e0 2024-09-21T01:58:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.381+0000 7f621e7fc640 1 -- 172.21.15.16:0/350393648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v16)=0 v16) ==== 108+0+0 (secure 0 0 0) 0x7f621801ad30 con 0x7f62281047e0 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.383+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 msgr2=0x7f6214042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.383+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 0x7f6214042080 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f622402a9a0 tx=0x7f6224005fd0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.383+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 msgr2=0x7f6228075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.383+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f621800d950 tx=0x7f621800de10 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 shutdown_connections 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f621403fb70 0x7f6214042080 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6228105180 0x7f6228075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62281047e0 0x7f6228075710 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 --2- 172.21.15.16:0/350393648 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6228103750 0x7f6228076e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 >> 172.21.15.16:0/350393648 conn(0x7f62280ff0e0 msgr2=0x7f6228107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 shutdown_connections 2024-09-21T01:58:00.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.384+0000 7f622f7b3640 1 -- 172.21.15.16:0/350393648 wait complete. 2024-09-21T01:58:00.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.566+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7fcf580ffb00 con 0x7fcf58106900 2024-09-21T01:58:00.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.570+0000 7fcf557fa640 1 -- 172.21.15.16:0/1780243820 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v17)=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fcf4c01c560 con 0x7fcf58106900 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 msgr2=0x7fcf40042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 0x7fcf40042080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcf48000c00 tx=0x7fcf48009560 comp rx=0 tx=0).stop 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 msgr2=0x7fcf5819a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00d950 tx=0x7fcf4c00de10 comp rx=0 tx=0).stop 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 shutdown_connections 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4003fb70 0x7fcf40042080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf581094a0 0x7fcf5819af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf58106900 0x7fcf5819a3c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 --2- 172.21.15.16:0/1780243820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf5806c340 0x7fcf58199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 >> 172.21.15.16:0/1780243820 conn(0x7fcf580fdfe0 msgr2=0x7fcf58104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 shutdown_connections 2024-09-21T01:58:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:00.572+0000 7fcf5e43c640 1 -- 172.21.15.16:0/1780243820 wait complete. 2024-09-21T01:58:00.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:00 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/772019936' entity='client.admin' 2024-09-21T01:58:00.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:00 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1967626546' entity='client.admin' 2024-09-21T01:58:00.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:00 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/350393648' entity='client.admin' 2024-09-21T01:58:00.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:00 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1780243820' entity='client.admin' 2024-09-21T01:58:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:00 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/772019936' entity='client.admin' 2024-09-21T01:58:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:00 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1967626546' entity='client.admin' 2024-09-21T01:58:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:00 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/350393648' entity='client.admin' 2024-09-21T01:58:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:00 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1780243820' entity='client.admin' 2024-09-21T01:58:01.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:00 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/772019936' entity='client.admin' 2024-09-21T01:58:01.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:00 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1967626546' entity='client.admin' 2024-09-21T01:58:01.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:00 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/350393648' entity='client.admin' 2024-09-21T01:58:01.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:00 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1780243820' entity='client.admin' 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 -- 172.21.15.16:0/2420182354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc102660 msgr2=0x7faadc102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 --2- 172.21.15.16:0/2420182354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc102660 0x7faadc102ad0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faacc009ca0 tx=0x7faacc02a950 comp rx=0 tx=0).stop 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 -- 172.21.15.16:0/2420182354 shutdown_connections 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 --2- 172.21.15.16:0/2420182354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc104090 0x7faadc10aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 --2- 172.21.15.16:0/2420182354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc1036f0 0x7faadc103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 --2- 172.21.15.16:0/2420182354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc102660 0x7faadc102ad0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 -- 172.21.15.16:0/2420182354 >> 172.21.15.16:0/2420182354 conn(0x7faadc0fdfd0 msgr2=0x7faadc100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 -- 172.21.15.16:0/2420182354 shutdown_connections 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.104+0000 7faae0e75640 1 -- 172.21.15.16:0/2420182354 wait complete. 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 Processor -- start 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 -- start start 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102660 0x7faadc199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 0x7faadc19b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faadc10dfa0 con 0x7faadc1036f0 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faadc10de20 con 0x7faadc102660 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.105+0000 7faae0e75640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faadc10e120 con 0x7faadc104090 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36114/0 (socket says 172.21.15.16:36114) 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 -- 172.21.15.16:0/3980776104 learned_addr learned my addr 172.21.15.16:0/3980776104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faada59c640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102660 0x7faadc199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faadad9d640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 0x7faadc19b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 -- 172.21.15.16:0/3980776104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 msgr2=0x7faadc19b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 0x7faadc19b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 -- 172.21.15.16:0/3980776104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102660 msgr2=0x7faadc199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102660 0x7faadc199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 -- 172.21.15.16:0/3980776104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faadc19b890 con 0x7faadc1036f0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faadad9d640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 0x7faadc19b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad9d9b640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faac400c970 tx=0x7faac400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4007bb0 con 0x7faadc1036f0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7faac4007d50 con 0x7faadc1036f0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faadc19bb80 con 0x7faadc1036f0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4004c80 con 0x7faadc1036f0 2024-09-21T01:58:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.106+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faadc1a2280 con 0x7faadc1036f0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.107+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7faac400f450 con 0x7faadc1036f0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.107+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faadc102ad0 con 0x7faadc1036f0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.108+0000 7faad37fe640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 0x7faac8042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.108+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7faac40544e0 con 0x7faadc1036f0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.108+0000 7faada59c640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 0x7faac8042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.108+0000 7faada59c640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 0x7faac8042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc009800 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.113+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faac4014030 con 0x7faadc1036f0 2024-09-21T01:58:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.304+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7faadc1a2570 con 0x7faadc1036f0 2024-09-21T01:58:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.308+0000 7faad37fe640 1 -- 172.21.15.16:0/3980776104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v18)=0 v18) ==== 120+0+0 (secure 0 0 0) 0x7faac4005590 con 0x7faadc1036f0 2024-09-21T01:58:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 msgr2=0x7faac8042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 0x7faac8042080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc009800 comp rx=0 tx=0).stop 2024-09-21T01:58:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 msgr2=0x7faadc19a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faac400c970 tx=0x7faac400ce30 comp rx=0 tx=0).stop 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 shutdown_connections 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faac803fb70 0x7faac8042080 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faadc104090 0x7faadc19b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1036f0 0x7faadc19a4d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 --2- 172.21.15.16:0/3980776104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102660 0x7faadc199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 >> 172.21.15.16:0/3980776104 conn(0x7faadc0fdfd0 msgr2=0x7faadc06bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 shutdown_connections 2024-09-21T01:58:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:01.310+0000 7faae0e75640 1 -- 172.21.15.16:0/3980776104 wait complete. 2024-09-21T01:58:02.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 -- 172.21.15.16:0/3002646968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc102640 msgr2=0x7f7ccc102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/3002646968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc102640 0x7f7ccc102ab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009900 tx=0x7f7cbc02a480 comp rx=0 tx=0).stop 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 -- 172.21.15.16:0/3002646968 shutdown_connections 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/3002646968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc104070 0x7f7ccc10aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/3002646968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc1036d0 0x7f7ccc103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/3002646968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc102640 0x7f7ccc102ab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 -- 172.21.15.16:0/3002646968 >> 172.21.15.16:0/3002646968 conn(0x7f7ccc0fdfb0 msgr2=0x7f7ccc100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 -- 172.21.15.16:0/3002646968 shutdown_connections 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.386+0000 7f7cd2d15640 1 -- 172.21.15.16:0/3002646968 wait complete. 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 Processor -- start 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 -- start start 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 0x7f7ccc199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:02.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc1036d0 0x7f7ccc19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10de30 con 0x7f7ccc104070 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10dcb0 con 0x7f7ccc102640 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd2d15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10dfb0 con 0x7f7ccc1036d0 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd12b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd12b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36036/0 (socket says 172.21.15.16:36036) 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.387+0000 7f7cd12b2640 1 -- 172.21.15.16:0/772019936 learned_addr learned my addr 172.21.15.16:0/772019936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7ccbfff640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc1036d0 0x7f7ccc19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd0ab1640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 0x7f7ccc199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 -- 172.21.15.16:0/772019936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc1036d0 msgr2=0x7f7ccc19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc1036d0 0x7f7ccc19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 -- 172.21.15.16:0/772019936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 msgr2=0x7f7ccc199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 0x7f7ccc199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 -- 172.21.15.16:0/772019936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19b6e0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd0ab1640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 0x7f7ccc199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd12b2640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400eaa0 tx=0x7f7cc400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400cdb0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19b9d0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(22 keys) ==== 976+0+0 (secure 0 0 0) 0x7f7cc4005020 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4011470 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.388+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc1a20d0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.389+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f7cc4011610 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.389+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc10e5f0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.390+0000 7f7cc9ffb640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 0x7f7cb8042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.390+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7cc40546f0 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.390+0000 7f7cd0ab1640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 0x7f7cb8042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.390+0000 7f7cd0ab1640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 0x7f7cb8042080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc005d50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.395+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7cc4016030 con 0x7f7ccc104070 2024-09-21T01:58:02.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.592+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f7ccc1a26e0 con 0x7f7ccc104070 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.595+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v14)=0 v14) ==== 114+0+0 (secure 0 0 0) 0x7f7cc400f360 con 0x7f7ccc104070 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.596+0000 7f7cc9ffb640 1 -- 172.21.15.16:0/772019936 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f7cc4021dc0 con 0x7f7ccc104070 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 msgr2=0x7f7cb8042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 0x7f7cb8042080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc005d50 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 msgr2=0x7f7ccc19aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400eaa0 tx=0x7f7cc400ef60 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 shutdown_connections 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7cb803fb70 0x7f7cb8042080 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ccc104070 0x7f7ccc19aec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ccc1036d0 0x7f7ccc19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 --2- 172.21.15.16:0/772019936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ccc102640 0x7f7ccc199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 >> 172.21.15.16:0/772019936 conn(0x7f7ccc0fdfb0 msgr2=0x7f7ccc0ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 shutdown_connections 2024-09-21T01:58:02.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:57:59.598+0000 7f7cd2d15640 1 -- 172.21.15.16:0/772019936 wait complete. 2024-09-21T01:58:02.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:02 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3980776104' entity='client.admin' 2024-09-21T01:58:02.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:02 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3980776104' entity='client.admin' 2024-09-21T01:58:02.764 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi016') 2024-09-21T01:58:02.764 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-09-21T01:58:02.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:02 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3980776104' entity='client.admin' 2024-09-21T01:58:04.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-09-21T01:58:04.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4126781391' entity='client.admin' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: Deploying daemon mgr.z on smithi161 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:03 smithi016 ceph-mon[31245]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.194 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.194 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.194 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-09-21T01:58:04.194 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.194 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4126781391' entity='client.admin' 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.195 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: Deploying daemon mgr.z on smithi161 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.196 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:03 smithi161 ceph-mon[34061]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"} : dispatch 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:04.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi016:/etc/ceph/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi123:/etc/ceph/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi161:/etc/ceph/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Updating smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4126781391' entity='client.admin' 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: Deploying daemon mgr.z on smithi161 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:03 smithi123 ceph-mon[33453]: from='client.? ' entity='client.admin' 2024-09-21T01:58:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 -- 172.21.15.16:0/981979723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719810de60 msgr2=0x7f7198110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 --2- 172.21.15.16:0/981979723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719810de60 0x7f7198110300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7190009ca0 tx=0x7f719002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 -- 172.21.15.16:0/981979723 shutdown_connections 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 --2- 172.21.15.16:0/981979723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f719810de60 0x7f7198110300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 --2- 172.21.15.16:0/981979723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719810d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 --2- 172.21.15.16:0/981979723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71981048b0 0x7f7198104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 -- 172.21.15.16:0/981979723 >> 172.21.15.16:0/981979723 conn(0x7f7198079730 msgr2=0x7f7198079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 -- 172.21.15.16:0/981979723 shutdown_connections 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.420+0000 7f719d44c640 1 -- 172.21.15.16:0/981979723 wait complete. 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 Processor -- start 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 -- start start 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f719810de60 0x7f719819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7198113800 con 0x7f71981048b0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7198113680 con 0x7f71981052e0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f719d44c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7198113980 con 0x7f719810de60 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36184/0 (socket says 172.21.15.16:36184) 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 -- 172.21.15.16:0/1449005871 learned_addr learned my addr 172.21.15.16:0/1449005871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f71967fc640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f71977fe640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f719810de60 0x7f719819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 -- 172.21.15.16:0/1449005871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f719810de60 msgr2=0x7f719819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f719810de60 0x7f719819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 -- 172.21.15.16:0/1449005871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 msgr2=0x7f719819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f7196ffd640 1 -- 172.21.15.16:0/1449005871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71981a0200 con 0x7f71981048b0 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.421+0000 7f71967fc640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f7196ffd640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f718800d950 tx=0x7f718800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188017070 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7188009c30 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71981a04f0 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718800f3c0 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.422+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71981a6c50 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.423+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f718801a070 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.423+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719806c120 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.423+0000 7f717ffff640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 0x7f7180042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.423+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7188054040 con 0x7f71981048b0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.423+0000 7f71967fc640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 0x7f7180042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.424+0000 7f71967fc640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 0x7f7180042080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71981023e0 tx=0x7f718c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.428+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7188009de0 con 0x7f71981048b0 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 -- 172.21.15.16:0/2139420451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff81036f0 msgr2=0x7f8ff8103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 --2- 172.21.15.16:0/2139420451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff81036f0 0x7f8ff8103b00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009900 tx=0x7f8fe802a480 comp rx=0 tx=0).stop 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 -- 172.21.15.16:0/2139420451 shutdown_connections 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 --2- 172.21.15.16:0/2139420451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 0x7f8ff810aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 --2- 172.21.15.16:0/2139420451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff81036f0 0x7f8ff8103b00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 --2- 172.21.15.16:0/2139420451 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff8102660 0x7f8ff8102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 -- 172.21.15.16:0/2139420451 >> 172.21.15.16:0/2139420451 conn(0x7f8ff80fdfd0 msgr2=0x7f8ff8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 -- 172.21.15.16:0/2139420451 shutdown_connections 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.355+0000 7f8fff898640 1 -- 172.21.15.16:0/2139420451 wait complete. 2024-09-21T01:58:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 Processor -- start 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 -- start start 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff81036f0 0x7f8ff819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 0x7f8ff819afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810dee0 con 0x7f8ff8102660 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810dd60 con 0x7f8ff8104090 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8fff898640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810e060 con 0x7f8ff81036f0 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8ffd634640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8ffd634640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36128/0 (socket says 172.21.15.16:36128) 2024-09-21T01:58:04.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8ffd634640 1 -- 172.21.15.16:0/4126781391 learned_addr learned my addr 172.21.15.16:0/4126781391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8ffce33640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff81036f0 0x7f8ff819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.356+0000 7f8ffde35640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 0x7f8ff819afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 -- 172.21.15.16:0/4126781391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff81036f0 msgr2=0x7f8ff819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff81036f0 0x7f8ff819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 -- 172.21.15.16:0/4126781391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 msgr2=0x7f8ff819afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 0x7f8ff819afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 -- 172.21.15.16:0/4126781391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff819b7f0 con 0x7f8ff8102660 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8ffd634640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e9f0 tx=0x7f8ff400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff400cd80 con 0x7f8ff8102660 2024-09-21T01:58:04.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.357+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f8ff4005470 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.358+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4004e60 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.358+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff819bae0 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.358+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a21e0 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.359+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8102ad0 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.359+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f8ff4005020 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.360+0000 7f8fee7fc640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 0x7f8fe4042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.360+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8ff4054af0 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.360+0000 7f8ffce33640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 0x7f8fe4042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.360+0000 7f8ffce33640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 0x7f8fe4042080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe8009560 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.365+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8ff4013190 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.567+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f8ff81a24d0 con 0x7f8ff8102660 2024-09-21T01:58:04.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.571+0000 7f8fee7fc640 1 -- 172.21.15.16:0/4126781391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v19)=0 v19) ==== 119+0+0 (secure 0 0 0) 0x7f8ff4019dc0 con 0x7f8ff8102660 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 msgr2=0x7f8fe4042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 0x7f8fe4042080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe8009560 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 msgr2=0x7f8ff8199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e9f0 tx=0x7f8ff400edf0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 shutdown_connections 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8fe403fb70 0x7f8fe4042080 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ff8104090 0x7f8ff819afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ff81036f0 0x7f8ff819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 --2- 172.21.15.16:0/4126781391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8102660 0x7f8ff8199dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 >> 172.21.15.16:0/4126781391 conn(0x7f8ff80fdfd0 msgr2=0x7f8ff80ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 shutdown_connections 2024-09-21T01:58:04.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f8fff898640 1 -- 172.21.15.16:0/4126781391 wait complete. 2024-09-21T01:58:04.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.620+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f719819b3c0 con 0x7f71981048b0 2024-09-21T01:58:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.623+0000 7f717ffff640 1 -- 172.21.15.16:0/1449005871 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v22)=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f719819b3c0 con 0x7f71981048b0 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 msgr2=0x7f7180042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 0x7f7180042080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71981023e0 tx=0x7f718c005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 msgr2=0x7f719819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f718800d950 tx=0x7f718800de10 comp rx=0 tx=0).stop 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 shutdown_connections 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f718003fb70 0x7f7180042080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f719810de60 0x7f719819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71981052e0 0x7f719819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 --2- 172.21.15.16:0/1449005871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71981048b0 0x7f719819a290 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 >> 172.21.15.16:0/1449005871 conn(0x7f7198079730 msgr2=0x7f7198100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 shutdown_connections 2024-09-21T01:58:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:04.626+0000 7f719d44c640 1 -- 172.21.15.16:0/1449005871 wait complete. 2024-09-21T01:58:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:05.415 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 systemd[1]: Starting Ceph mgr.z for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T01:58:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 -- 172.21.15.16:0/4208274746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a0104070 msgr2=0x7fa5a010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 --2- 172.21.15.16:0/4208274746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a0104070 0x7fa5a010aa10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa598009ca0 tx=0x7fa59802a950 comp rx=0 tx=0).stop 2024-09-21T01:58:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 -- 172.21.15.16:0/4208274746 shutdown_connections 2024-09-21T01:58:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 --2- 172.21.15.16:0/4208274746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a0104070 0x7fa5a010aa10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 --2- 172.21.15.16:0/4208274746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a01036d0 0x7fa5a0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 --2- 172.21.15.16:0/4208274746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 0x7fa5a0102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 -- 172.21.15.16:0/4208274746 >> 172.21.15.16:0/4208274746 conn(0x7fa5a00fdfb0 msgr2=0x7fa5a0100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 -- 172.21.15.16:0/4208274746 shutdown_connections 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.482+0000 7fa5a8352640 1 -- 172.21.15.16:0/4208274746 wait complete. 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 Processor -- start 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 -- start start 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 0x7fa5a019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 0x7fa5a019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 0x7fa5a019d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010df10 con 0x7fa5a01036d0 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010dd90 con 0x7fa5a0104070 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.483+0000 7fa5a8352640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010e090 con 0x7fa5a0102640 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a58ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 0x7fa5a019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a58ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 0x7fa5a019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36142/0 (socket says 172.21.15.16:36142) 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a58ed640 1 -- 172.21.15.16:0/2200941116 learned_addr learned my addr 172.21.15.16:0/2200941116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 0x7fa5a019d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a60ee640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 0x7fa5a019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 -- 172.21.15.16:0/2200941116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 msgr2=0x7fa5a019bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 0x7fa5a019bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 -- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 msgr2=0x7fa5a019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 0x7fa5a019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 -- 172.21.15.16:0/2200941116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a019d920 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.484+0000 7fa5a68ef640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 0x7fa5a019d100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa59802af40 tx=0x7fa598003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.485+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598034040 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.485+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7fa59802fbd0 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.485+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598004370 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.485+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a019dbb0 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.485+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a01a4140 con 0x7fa5a0104070 2024-09-21T01:58:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.486+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7fa598037020 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.486+0000 7fa5977fe640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 0x7fa58c042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.486+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a0102ab0 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.486+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa598075490 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.487+0000 7fa5a60ee640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 0x7fa58c042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.487+0000 7fa5a60ee640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 0x7fa58c042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009c20 tx=0x7fa59c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.492+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa59802edf0 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.734+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fa5a01a4430 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.741+0000 7fa5977fe640 1 -- 172.21.15.16:0/2200941116 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v20)=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fa598046da0 con 0x7fa5a0104070 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 msgr2=0x7fa58c042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 0x7fa58c042080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009c20 tx=0x7fa59c005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 msgr2=0x7fa5a019d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 0x7fa5a019d100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa59802af40 tx=0x7fa598003b80 comp rx=0 tx=0).stop 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 shutdown_connections 2024-09-21T01:58:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa58c03fb70 0x7fa58c042080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a0104070 0x7fa5a019d100 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5a01036d0 0x7fa5a019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 --2- 172.21.15.16:0/2200941116 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5a0102640 0x7fa5a019bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 >> 172.21.15.16:0/2200941116 conn(0x7fa5a00fdfb0 msgr2=0x7fa5a00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 shutdown_connections 2024-09-21T01:58:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.743+0000 7fa5a8352640 1 -- 172.21.15.16:0/2200941116 wait complete. 2024-09-21T01:58:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:05 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1449005871' entity='client.admin' 2024-09-21T01:58:05.779 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:05 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1449005871' entity='client.admin' 2024-09-21T01:58:05.779 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 podman[36017]: 2024-09-21 01:58:05.429337712 +0000 UTC m=+0.015579064 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:58:05.779 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 podman[36017]: 2024-09-21 01:58:05.532363741 +0000 UTC m=+0.118605083 container create 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T01:58:05.779 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 podman[36017]: 2024-09-21 01:58:05.776668366 +0000 UTC m=+0.362909713 container init 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T01:58:05.779 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 podman[36017]: 2024-09-21 01:58:05.779400402 +0000 UTC m=+0.365641747 container start 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T01:58:05.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:05 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1449005871' entity='client.admin' 2024-09-21T01:58:06.164 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 ceph-mgr[36036]: -- 172.21.15.161:0/1017379524 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6418470e0 con 0x55a641865a80 2024-09-21T01:58:06.165 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 bash[36017]: 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 2024-09-21T01:58:06.165 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 01:58:05 smithi161 systemd[1]: Started Ceph mgr.z for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:58:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 -- 172.21.15.16:0/802598585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8106950 msgr2=0x7f7dd8108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 --2- 172.21.15.16:0/802598585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8106950 0x7f7dd8108d70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009900 tx=0x7f7dc802a410 comp rx=0 tx=0).stop 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 -- 172.21.15.16:0/802598585 shutdown_connections 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 --2- 172.21.15.16:0/802598585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd8109490 0x7f7dd810b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 --2- 172.21.15.16:0/802598585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8106950 0x7f7dd8108d70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 --2- 172.21.15.16:0/802598585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd806b900 0x7f7dd8106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 -- 172.21.15.16:0/802598585 >> 172.21.15.16:0/802598585 conn(0x7f7dd80fdfb0 msgr2=0x7f7dd8100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 -- 172.21.15.16:0/802598585 shutdown_connections 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.575+0000 7f7de03d2640 1 -- 172.21.15.16:0/802598585 wait complete. 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f7de03d2640 1 Processor -- start 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.576+0000 7f7de03d2640 1 -- start start 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7de03d2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 0x7f7dd8199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7de03d2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7de03d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 0x7f7dd819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7dde16e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 0x7f7dd8199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7de03d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd810f000 con 0x7f7dd8109490 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7ddd96d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7ddd96d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:57250/0 (socket says 172.21.15.16:57250) 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7ddd96d640 1 -- 172.21.15.16:0/1410032280 learned_addr learned my addr 172.21.15.16:0/1410032280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.577+0000 7f7dde96f640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 0x7f7dd819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd810ee80 con 0x7f7dd8106950 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd810f180 con 0x7f7dd806b900 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7ddd96d640 1 -- 172.21.15.16:0/1410032280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 msgr2=0x7f7dd8199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7ddd96d640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 0x7f7dd8199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7ddd96d640 1 -- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 msgr2=0x7f7dd819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7ddd96d640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 0x7f7dd819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7ddd96d640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd819b550 con 0x7f7dd8106950 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.578+0000 7f7dde96f640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 0x7f7dd819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.579+0000 7f7dde16e640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 0x7f7dd8199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.579+0000 7f7ddd96d640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009850 tx=0x7f7dc8003ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.579+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8035030 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.579+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f7dc8041070 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.579+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8031d10 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.580+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd819b7e0 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.580+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd81021f0 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.581+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd810f7c0 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.582+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f7dc803db70 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.582+0000 7f7dcf7fe640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 0x7f7dc4042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.583+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7dc8079110 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.583+0000 7f7dde16e640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 0x7f7dc4042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.583+0000 7f7dde16e640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 0x7f7dc4042080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009cd0 tx=0x7f7dd4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.586+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7dc8041220 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.793+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f7dd8063fc0 con 0x7f7dd8106950 2024-09-21T01:58:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.800+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 7 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7dc800b0a0 con 0x7f7dd8106950 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.800+0000 7f7dcf7fe640 1 -- 172.21.15.16:0/1410032280 <== mon.2 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v21)=0 v21) ==== 123+0+0 (secure 0 0 0) 0x7f7dc8031590 con 0x7f7dd8106950 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 msgr2=0x7f7dc4042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 0x7f7dc4042080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009cd0 tx=0x7f7dd4005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 msgr2=0x7f7dd819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8009850 tx=0x7f7dc8003ed0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 shutdown_connections 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dc403fb70 0x7f7dc4042080 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dd8109490 0x7f7dd819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd8106950 0x7f7dd819a190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 --2- 172.21.15.16:0/1410032280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dd806b900 0x7f7dd8199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 >> 172.21.15.16:0/1410032280 conn(0x7f7dd80fdfb0 msgr2=0x7f7dd8107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 shutdown_connections 2024-09-21T01:58:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:03.802+0000 7f7de03d2640 1 -- 172.21.15.16:0/1410032280 wait complete. 2024-09-21T01:58:06.738 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-09-21T01:58:07.160 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:07.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:07.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:07.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:07.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:06 smithi123 ceph-mon[33453]: Deploying daemon mgr.y on smithi123 2024-09-21T01:58:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:07.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:07.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:07.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:06 smithi016 ceph-mon[31245]: Deploying daemon mgr.y on smithi123 2024-09-21T01:58:07.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:06 smithi161 ceph-mon[34061]: Deploying daemon mgr.y on smithi123 2024-09-21T01:58:07.610 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:07 smithi123 systemd[1]: Starting Ceph mgr.y for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T01:58:07.929 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:07 smithi123 podman[34972]: 2024-09-21 01:58:07.724107799 +0000 UTC m=+0.016094580 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:58:07.929 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:07 smithi123 podman[34972]: 2024-09-21 01:58:07.832114017 +0000 UTC m=+0.124100789 container create caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T01:58:08.270 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:08 smithi123 podman[34972]: 2024-09-21 01:58:08.042585928 +0000 UTC m=+0.334572702 container init caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:58:08.270 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:08 smithi123 podman[34972]: 2024-09-21 01:58:08.046072777 +0000 UTC m=+0.338059550 container start caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:58:08.270 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:08 smithi123 ceph-mgr[34991]: -- 172.21.15.123:0/322328838 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d333e70e0 con 0x555d33405a80 2024-09-21T01:58:08.271 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:08 smithi123 bash[34972]: caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d 2024-09-21T01:58:08.271 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 01:58:08 smithi123 systemd[1]: Started Ceph mgr.y for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T01:58:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.304+0000 7f9695879640 1 -- 172.21.15.16:0/411580306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 msgr2=0x7f969010f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 --2- 172.21.15.16:0/411580306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969010f530 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9688009ca0 tx=0x7f968802a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 -- 172.21.15.16:0/411580306 shutdown_connections 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 --2- 172.21.15.16:0/411580306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969010f530 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 --2- 172.21.15.16:0/411580306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96901083b0 0x7f96901087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 --2- 172.21.15.16:0/411580306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9690102a70 0x7f9690102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 -- 172.21.15.16:0/411580306 >> 172.21.15.16:0/411580306 conn(0x7f96900796e0 msgr2=0x7f9690079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 -- 172.21.15.16:0/411580306 shutdown_connections 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 -- 172.21.15.16:0/411580306 wait complete. 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.305+0000 7f9695879640 1 Processor -- start 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 -- start start 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9690102a70 0x7f969019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96901083b0 0x7f969019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112750 con 0x7f9690108d00 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96901125d0 con 0x7f9690102a70 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f9695879640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96901128d0 con 0x7f96901083b0 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50768/0 (socket says 172.21.15.16:50768) 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 -- 172.21.15.16:0/1983872178 learned_addr learned my addr 172.21.15.16:0/1983872178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968effd640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9690102a70 0x7f969019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968e7fc640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96901083b0 0x7f969019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 -- 172.21.15.16:0/1983872178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96901083b0 msgr2=0x7f969019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96901083b0 0x7f969019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 -- 172.21.15.16:0/1983872178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9690102a70 msgr2=0x7f969019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9690102a70 0x7f969019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 -- 172.21.15.16:0/1983872178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96901a0280 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.306+0000 7f968f7fe640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9688005cc0 tx=0x7f96880093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.307+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688003cf0 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.307+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f96880044b0 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.307+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96901a0510 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.307+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688030a70 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.307+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901a6cd0 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.308+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9690103cf0 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.308+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f9688030c70 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.308+0000 7f9694877640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 0x7f9678042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.308+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f96880759c0 con 0x7f9690108d00 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.308+0000 7f968effd640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 0x7f9678042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.309+0000 7f968effd640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 0x7f9678042080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9680005d00 tx=0x7f968000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.313+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9688004660 con 0x7f9690108d00 2024-09-21T01:58:08.373 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.526+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f969010b570 con 0x7f9690108d00 2024-09-21T01:58:08.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.529+0000 7f9694877640 1 -- 172.21.15.16:0/1983872178 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v23)=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f968803b410 con 0x7f9690108d00 2024-09-21T01:58:08.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.531+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 msgr2=0x7f9678042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.531+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 0x7f9678042080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9680005d00 tx=0x7f968000b410 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.531+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 msgr2=0x7f969019fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.531+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9688005cc0 tx=0x7f96880093e0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 shutdown_connections 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f967803fb70 0x7f9678042080 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9690108d00 0x7f969019fb50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96901083b0 0x7f969019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 --2- 172.21.15.16:0/1983872178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9690102a70 0x7f969019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 >> 172.21.15.16:0/1983872178 conn(0x7f96900796e0 msgr2=0x7f969006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 shutdown_connections 2024-09-21T01:58:08.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:08.532+0000 7f9695879640 1 -- 172.21.15.16:0/1983872178 wait complete. 2024-09-21T01:58:08.558 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:09.486 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.486 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:09.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:09 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1983872178' entity='client.admin' 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:09 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1983872178' entity='client.admin' 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.582+0000 7f4e88556640 1 -- 172.21.15.16:0/2838390101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e801062b0 msgr2=0x7f4e801066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.582+0000 7f4e88556640 1 --2- 172.21.15.16:0/2838390101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e801062b0 0x7f4e801066c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009900 tx=0x7f4e7002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 -- 172.21.15.16:0/2838390101 shutdown_connections 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 --2- 172.21.15.16:0/2838390101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e80110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 --2- 172.21.15.16:0/2838390101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e801062b0 0x7f4e801066c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 --2- 172.21.15.16:0/2838390101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80102060 0x7f4e80102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 -- 172.21.15.16:0/2838390101 >> 172.21.15.16:0/2838390101 conn(0x7f4e80077cc0 msgr2=0x7f4e80078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 -- 172.21.15.16:0/2838390101 shutdown_connections 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 -- 172.21.15.16:0/2838390101 wait complete. 2024-09-21T01:58:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 Processor -- start 2024-09-21T01:58:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.583+0000 7f4e88556640 1 -- start start 2024-09-21T01:58:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 0x7f4e8019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e8019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e80113860 con 0x7f4e80102060 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e801136e0 con 0x7f4e801062b0 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e88556640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e801139e0 con 0x7f4e80106c00 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e862f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 0x7f4e8019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e862f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50784/0 (socket says 172.21.15.16:50784) 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e862f2640 1 -- 172.21.15.16:0/3252325196 learned_addr learned my addr 172.21.15.16:0/3252325196 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e86af3640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e8019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 -- 172.21.15.16:0/3252325196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 msgr2=0x7f4e8019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e8019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 -- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 msgr2=0x7f4e8019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e85af1640 1 -- 172.21.15.16:0/3252325196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e801a0260 con 0x7f4e801062b0 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e862f2640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.584+0000 7f4e86af3640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e8019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e85af1640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 0x7f4e8019a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e7002a930 tx=0x7f4e70005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7003f030 con 0x7f4e801062b0 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f4e70032c70 con 0x7f4e801062b0 2024-09-21T01:58:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e801a04f0 con 0x7f4e801062b0 2024-09-21T01:58:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70035070 con 0x7f4e801062b0 2024-09-21T01:58:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.585+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e801a6cb0 con 0x7f4e801062b0 2024-09-21T01:58:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.586+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8006b8b0 con 0x7f4e801062b0 2024-09-21T01:58:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.586+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f4e70042030 con 0x7f4e801062b0 2024-09-21T01:58:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.587+0000 7f4e777fe640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 0x7f4e6c042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.587+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4e700749e0 con 0x7f4e801062b0 2024-09-21T01:58:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.587+0000 7f4e862f2640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 0x7f4e6c042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.587+0000 7f4e862f2640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 0x7f4e6c042080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0058c0 tx=0x7f4e7c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.591+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4e70039df0 con 0x7f4e801062b0 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:09 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1983872178' entity='client.admin' 2024-09-21T01:58:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.783+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f4e8010bc90 con 0x7f4e801062b0 2024-09-21T01:58:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.790+0000 7f4e777fe640 1 -- 172.21.15.16:0/3252325196 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v24)=0 v24) ==== 125+0+0 (secure 0 0 0) 0x7f4e70003af0 con 0x7f4e801062b0 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.792+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 msgr2=0x7f4e6c042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.792+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 0x7f4e6c042080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0058c0 tx=0x7f4e7c005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.792+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 msgr2=0x7f4e8019a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.792+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 0x7f4e8019a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e7002a930 tx=0x7f4e70005e00 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 shutdown_connections 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e6c03fb70 0x7f4e6c042080 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e80106c00 0x7f4e8019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e801062b0 0x7f4e8019a890 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 --2- 172.21.15.16:0/3252325196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e80102060 0x7f4e8019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 >> 172.21.15.16:0/3252325196 conn(0x7f4e80077cc0 msgr2=0x7f4e801075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 shutdown_connections 2024-09-21T01:58:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:09.793+0000 7f4e88556640 1 -- 172.21.15.16:0/3252325196 wait complete. 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: Reconfiguring daemon mgr.x on smithi016 2024-09-21T01:58:10.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:10 smithi016 ceph-mon[31245]: from='client.? ' entity='client.admin' 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- 172.21.15.16:0/4255756888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108d70 msgr2=0x7f100810f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 --2- 172.21.15.16:0/4255756888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108d70 0x7f100810f5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1000009ca0 tx=0x7f100002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- 172.21.15.16:0/4255756888 shutdown_connections 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 --2- 172.21.15.16:0/4255756888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108d70 0x7f100810f5a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 --2- 172.21.15.16:0/4255756888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008108420 0x7f1008108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 --2- 172.21.15.16:0/4255756888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008102ae0 0x7f1008102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- 172.21.15.16:0/4255756888 >> 172.21.15.16:0/4255756888 conn(0x7f1008079770 msgr2=0x7f1008079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- 172.21.15.16:0/4255756888 shutdown_connections 2024-09-21T01:58:10.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- 172.21.15.16:0/4255756888 wait complete. 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 Processor -- start 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.462+0000 7f100ed5b640 1 -- start start 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 0x7f100819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008108d70 0x7f100819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1008112810 con 0x7f1008108420 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1008112690 con 0x7f1008102ae0 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100ed5b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1008112990 con 0x7f1008108d70 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 0x7f100819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100caf7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100caf7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:56548/0 (socket says 172.21.15.16:56548) 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100caf7640 1 -- 172.21.15.16:0/969130995 learned_addr learned my addr 172.21.15.16:0/969130995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100d2f8640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008108d70 0x7f100819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 -- 172.21.15.16:0/969130995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008108d70 msgr2=0x7f100819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008108d70 0x7f100819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 -- 172.21.15.16:0/969130995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 msgr2=0x7f100819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 -- 172.21.15.16:0/969130995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10081a0280 con 0x7f1008108420 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f100caf7640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.463+0000 7f1007fff640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 0x7f100819a8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f100806b9c0 tx=0x7f0ffc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.464+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00cc40 con 0x7f1008108420 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.464+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f0ffc00cde0 con 0x7f1008108420 2024-09-21T01:58:10.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.464+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10081a0570 con 0x7f1008108420 2024-09-21T01:58:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.464+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc018680 con 0x7f1008108420 2024-09-21T01:58:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.464+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10081a6cd0 con 0x7f1008108420 2024-09-21T01:58:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.465+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f0ffc018820 con 0x7f1008108420 2024-09-21T01:58:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.465+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1008103d60 con 0x7f1008108420 2024-09-21T01:58:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.465+0000 7f1005ffb640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 0x7f0ff0042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.465+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f0ffc054a70 con 0x7f1008108420 2024-09-21T01:58:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.465+0000 7f100caf7640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 0x7f0ff0042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.466+0000 7f100caf7640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 0x7f0ff0042080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8011550 tx=0x7f0ff8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.470+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0ffc01d210 con 0x7f1008108420 2024-09-21T01:58:10.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: Reconfiguring daemon mgr.x on smithi016 2024-09-21T01:58:10.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:10 smithi123 ceph-mon[33453]: from='client.? ' entity='client.admin' 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 -- 172.21.15.16:0/1225890220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6650102ad0 msgr2=0x7f6650102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 --2- 172.21.15.16:0/1225890220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6650102ad0 0x7f6650102ee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6644009900 tx=0x7f664402a410 comp rx=0 tx=0).stop 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 -- 172.21.15.16:0/1225890220 shutdown_connections 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 --2- 172.21.15.16:0/1225890220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 --2- 172.21.15.16:0/1225890220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66501083c0 0x7f66501087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 --2- 172.21.15.16:0/1225890220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6650102ad0 0x7f6650102ee0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 -- 172.21.15.16:0/1225890220 >> 172.21.15.16:0/1225890220 conn(0x7f66500796d0 msgr2=0x7f6650079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.646+0000 7f6656c52640 1 -- 172.21.15.16:0/1225890220 shutdown_connections 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.647+0000 7f6656c52640 1 -- 172.21.15.16:0/1225890220 wait complete. 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.647+0000 7f6656c52640 1 Processor -- start 2024-09-21T01:58:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.647+0000 7f6656c52640 1 -- start start 2024-09-21T01:58:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 0x7f665019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6650112760 con 0x7f66501083c0 2024-09-21T01:58:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66501125e0 con 0x7f6650108d10 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f6656c52640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66501128e0 con 0x7f6650102ad0 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50842/0 (socket says 172.21.15.16:50842) 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 -- 172.21.15.16:0/745566861 learned_addr learned my addr 172.21.15.16:0/745566861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f66549ee640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 0x7f665019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f66551ef640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 -- 172.21.15.16:0/745566861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 msgr2=0x7f665019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 0x7f665019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 -- 172.21.15.16:0/745566861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 msgr2=0x7f665019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 -- 172.21.15.16:0/745566861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66501a0370 con 0x7f66501083c0 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f66549ee640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 0x7f665019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f66551ef640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.648+0000 7f664ffff640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9c0 tx=0x7f663c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.649+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c00ccc0 con 0x7f66501083c0 2024-09-21T01:58:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.649+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66501a0660 con 0x7f66501083c0 2024-09-21T01:58:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.649+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f663c005490 con 0x7f66501083c0 2024-09-21T01:58:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.649+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c010430 con 0x7f66501083c0 2024-09-21T01:58:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.649+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66500ff580 con 0x7f66501083c0 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.650+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f663c0105d0 con 0x7f66501083c0 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.650+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665006b390 con 0x7f66501083c0 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.650+0000 7f664dffb640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 0x7f6640042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.650+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f663c054ac0 con 0x7f66501083c0 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.650+0000 7f66549ee640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 0x7f6640042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.651+0000 7f66549ee640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 0x7f6640042080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6644009230 tx=0x7f66440058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.655+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f663c016330 con 0x7f66501083c0 2024-09-21T01:58:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.660+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f10081a6f30 con 0x7f1008108420 2024-09-21T01:58:10.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: Reconfiguring mgr.x (unknown last config time)... 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: Reconfiguring daemon mgr.x on smithi016 2024-09-21T01:58:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:10 smithi161 ceph-mon[34061]: from='client.? ' entity='client.admin' 2024-09-21T01:58:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.664+0000 7f1005ffb640 1 -- 172.21.15.16:0/969130995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v25)=0 v25) ==== 120+0+0 (secure 0 0 0) 0x7f0ffc010a50 con 0x7f1008108420 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 msgr2=0x7f0ff0042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 0x7f0ff0042080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8011550 tx=0x7f0ff8005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 msgr2=0x7f100819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 0x7f100819a8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f100806b9c0 tx=0x7f0ffc00ee50 comp rx=0 tx=0).stop 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 shutdown_connections 2024-09-21T01:58:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ff003fb70 0x7f0ff0042080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1008108d70 0x7f100819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1008108420 0x7f100819a8b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 --2- 172.21.15.16:0/969130995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1008102ae0 0x7f100819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 >> 172.21.15.16:0/969130995 conn(0x7f1008079770 msgr2=0x7f1008106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 shutdown_connections 2024-09-21T01:58:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.667+0000 7f100ed5b640 1 -- 172.21.15.16:0/969130995 wait complete. 2024-09-21T01:58:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.855+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f665010baa0 con 0x7f66501083c0 2024-09-21T01:58:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.859+0000 7f664dffb640 1 -- 172.21.15.16:0/745566861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v26)=0 v26) ==== 116+0+0 (secure 0 0 0) 0x7f663c0108c0 con 0x7f66501083c0 2024-09-21T01:58:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 msgr2=0x7f6640042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 0x7f6640042080 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6644009230 tx=0x7f66440058d0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 msgr2=0x7f665019a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9c0 tx=0x7f663c00ee80 comp rx=0 tx=0).stop 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 shutdown_connections 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f664003fb70 0x7f6640042080 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6650108d10 0x7f665019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66501083c0 0x7f665019a9a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 --2- 172.21.15.16:0/745566861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6650102ad0 0x7f665019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 >> 172.21.15.16:0/745566861 conn(0x7f66500796d0 msgr2=0x7f6650101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.861+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 shutdown_connections 2024-09-21T01:58:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.862+0000 7f6656c52640 1 -- 172.21.15.16:0/745566861 wait complete. 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 -- 172.21.15.16:0/2157615713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 msgr2=0x7f2c30102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/2157615713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c30102e70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009900 tx=0x7f2c2002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 -- 172.21.15.16:0/2157615713 shutdown_connections 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/2157615713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 0x7f2c3010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/2157615713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c301087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/2157615713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c30102e70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 -- 172.21.15.16:0/2157615713 >> 172.21.15.16:0/2157615713 conn(0x7f2c30079730 msgr2=0x7f2c30079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 -- 172.21.15.16:0/2157615713 shutdown_connections 2024-09-21T01:58:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.868+0000 7f2c35ce7640 1 -- 172.21.15.16:0/2157615713 wait complete. 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.869+0000 7f2c35ce7640 1 Processor -- start 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.869+0000 7f2c35ce7640 1 -- start start 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c3019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 0x7f2c3019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30113810 con 0x7f2c30102a60 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30113690 con 0x7f2c30108cf0 2024-09-21T01:58:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c35ce7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30113990 con 0x7f2c301083a0 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50860/0 (socket says 172.21.15.16:50860) 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 -- 172.21.15.16:0/1418094081 learned_addr learned my addr 172.21.15.16:0/1418094081 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2effd640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c3019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2ffff640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 0x7f2c3019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 -- 172.21.15.16:0/1418094081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 msgr2=0x7f2c3019a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c3019a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 -- 172.21.15.16:0/1418094081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 msgr2=0x7f2c3019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 0x7f2c3019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2f7fe640 1 -- 172.21.15.16:0/1418094081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c301a0310 con 0x7f2c30102a60 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.870+0000 7f2c2effd640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c3019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c2f7fe640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006010 tx=0x7f2c20003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2003f070 con 0x7f2c30102a60 2024-09-21T01:58:10.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c301a05a0 con 0x7f2c30102a60 2024-09-21T01:58:10.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f2c2003bae0 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20035030 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.871+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c301a6d60 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.872+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 51915+0+0 (secure 0 0 0) 0x7f2c20042070 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.872+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c30102e70 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.872+0000 7f2c2cff9640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 0x7f2c18042080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.872+0000 7f2c2effd640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 0x7f2c18042080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.873+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2c20075070 con 0x7f2c30102a60 2024-09-21T01:58:10.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.873+0000 7f2c2effd640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 0x7f2c18042080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c30079fa0 tx=0x7f2c24005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:10.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:10.878+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2c200303a0 con 0x7f2c30102a60 2024-09-21T01:58:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.068+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f2c3019b4d0 con 0x7f2c30102a60 2024-09-21T01:58:11.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.071+0000 7f2c2cff9640 1 -- 172.21.15.16:0/1418094081 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v27)=0 v27) ==== 116+0+0 (secure 0 0 0) 0x7f2c20036260 con 0x7f2c30102a60 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 msgr2=0x7f2c18042080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 0x7f2c18042080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c30079fa0 tx=0x7f2c24005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 msgr2=0x7f2c3019a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2c20006010 tx=0x7f2c20003c10 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 shutdown_connections 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c1803fb70 0x7f2c18042080 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c30108cf0 0x7f2c3019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c301083a0 0x7f2c3019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 --2- 172.21.15.16:0/1418094081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c30102a60 0x7f2c3019a3a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 >> 172.21.15.16:0/1418094081 conn(0x7f2c30079730 msgr2=0x7f2c30101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 shutdown_connections 2024-09-21T01:58:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:11.074+0000 7f2c35ce7640 1 -- 172.21.15.16:0/1418094081 wait complete. 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/969130995' entity='client.admin' 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: Standby manager daemon z started 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/745566861' entity='client.admin' 2024-09-21T01:58:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1418094081' entity='client.admin' 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/969130995' entity='client.admin' 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: Standby manager daemon z started 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/745566861' entity='client.admin' 2024-09-21T01:58:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1418094081' entity='client.admin' 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/969130995' entity='client.admin' 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: Standby manager daemon z started 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/crt"} : dispatch 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/z/key"} : dispatch 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.161:0/485764332' entity='mgr.z' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/745566861' entity='client.admin' 2024-09-21T01:58:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1418094081' entity='client.admin' 2024-09-21T01:58:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 -- 172.21.15.16:0/3770907965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc628104840 msgr2=0x7fc628104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3770907965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc628104840 0x7fc628104c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc620009900 tx=0x7fc62002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 -- 172.21.15.16:0/3770907965 shutdown_connections 2024-09-21T01:58:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3770907965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc62810ddf0 0x7fc628110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3770907965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 0x7fc62810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3770907965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc628104840 0x7fc628104c50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 -- 172.21.15.16:0/3770907965 >> 172.21.15.16:0/3770907965 conn(0x7fc628079730 msgr2=0x7fc628079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 -- 172.21.15.16:0/3770907965 shutdown_connections 2024-09-21T01:58:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.324+0000 7fc62ca56640 1 -- 172.21.15.16:0/3770907965 wait complete. 2024-09-21T01:58:12.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.325+0000 7fc62ca56640 1 Processor -- start 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.325+0000 7fc62ca56640 1 -- start start 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628104840 0x7fc62819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 0x7fc62819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc628113850 con 0x7fc62810ddf0 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6281136d0 con 0x7fc628105270 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62ca56640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc6281139d0 con 0x7fc628104840 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50880/0 (socket says 172.21.15.16:50880) 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 -- 172.21.15.16:0/3424863002 learned_addr learned my addr 172.21.15.16:0/3424863002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc62659c640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628104840 0x7fc62819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc625d9b640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 0x7fc62819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:12.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 -- 172.21.15.16:0/3424863002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628104840 msgr2=0x7fc62819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628104840 0x7fc62819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 -- 172.21.15.16:0/3424863002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 msgr2=0x7fc62819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 0x7fc62819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.326+0000 7fc626d9d640 1 -- 172.21.15.16:0/3424863002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6281a02f0 con 0x7fc62810ddf0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc626d9d640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc61800eaa0 tx=0x7fc61800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61800cdb0 con 0x7fc62810ddf0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc618005020 con 0x7fc62810ddf0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6281a05e0 con 0x7fc62810ddf0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6180116d0 con 0x7fc62810ddf0 2024-09-21T01:58:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.327+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6281a6d40 con 0x7fc62810ddf0 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.328+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc628104c50 con 0x7fc62810ddf0 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.329+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 103194+0+0 (secure 0 0 0) 0x7fc618011870 con 0x7fc62810ddf0 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.329+0000 7fc61f7fe640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 0x7fc61407d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.329+0000 7fc62659c640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 0x7fc61407d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.329+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc61809cee0 con 0x7fc62810ddf0 2024-09-21T01:58:12.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.330+0000 7fc62659c640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 0x7fc61407d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc62002aa00 tx=0x7fc620005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:12.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.334+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc618016030 con 0x7fc62810ddf0 2024-09-21T01:58:12.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.534+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fc628108fd0 con 0x7fc62810ddf0 2024-09-21T01:58:12.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.537+0000 7fc61f7fe640 1 -- 172.21.15.16:0/3424863002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v28)=0 v28) ==== 106+0+0 (secure 0 0 0) 0x7fc6180698f0 con 0x7fc62810ddf0 2024-09-21T01:58:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 msgr2=0x7fc61407d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:12.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 0x7fc61407d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc62002aa00 tx=0x7fc620005a20 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 msgr2=0x7fc62819fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc61800eaa0 tx=0x7fc61800ef60 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 shutdown_connections 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc61407b020 0x7fc61407d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc62810ddf0 0x7fc62819fbc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc628105270 0x7fc62819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 --2- 172.21.15.16:0/3424863002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc628104840 0x7fc62819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 >> 172.21.15.16:0/3424863002 conn(0x7fc628079730 msgr2=0x7fc6280ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 shutdown_connections 2024-09-21T01:58:12.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:12.540+0000 7fc62ca56640 1 -- 172.21.15.16:0/3424863002 wait complete. 2024-09-21T01:58:12.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:12 smithi016 ceph-mon[31245]: mgrmap e18: x(active, since 19s), standbys: z 2024-09-21T01:58:12.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:12 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T01:58:12.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:12 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:12.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:12 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3424863002' entity='client.admin' 2024-09-21T01:58:13.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:12 smithi123 ceph-mon[33453]: mgrmap e18: x(active, since 19s), standbys: z 2024-09-21T01:58:13.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:12 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T01:58:13.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:12 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:13.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:12 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3424863002' entity='client.admin' 2024-09-21T01:58:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:12 smithi161 ceph-mon[34061]: mgrmap e18: x(active, since 19s), standbys: z 2024-09-21T01:58:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:12 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "z", "id": "z"} : dispatch 2024-09-21T01:58:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:12 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:12 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3424863002' entity='client.admin' 2024-09-21T01:58:13.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:13.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: Standby manager daemon y started 2024-09-21T01:58:13.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T01:58:13.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:13.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T01:58:13.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:13 smithi016 ceph-mon[31245]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: Standby manager daemon y started 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T01:58:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:13 smithi123 ceph-mon[33453]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: Standby manager daemon y started 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/crt"} : dispatch 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/y/key"} : dispatch 2024-09-21T01:58:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:13 smithi161 ceph-mon[34061]: from='mgr.? 172.21.15.123:0/1576543660' entity='mgr.y' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:58:14.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:14 smithi016 ceph-mon[31245]: mgrmap e19: x(active, since 21s), standbys: z, y 2024-09-21T01:58:14.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:14 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T01:58:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:14 smithi123 ceph-mon[33453]: mgrmap e19: x(active, since 21s), standbys: z, y 2024-09-21T01:58:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:14 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T01:58:15.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:14 smithi161 ceph-mon[34061]: mgrmap e19: x(active, since 21s), standbys: z, y 2024-09-21T01:58:15.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:14 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T01:58:15.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.444 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.444 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.454 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.456 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.468 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.468 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.469 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:58:15.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:15.982 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:15.982 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:15.982 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:15.982 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:58:15.983 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:58:15.983 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:58:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 -- 172.21.15.16:0/574319861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc1094e0 msgr2=0x7fd5fc1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 --2- 172.21.15.16:0/574319861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc1098f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009900 tx=0x7fd5f002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 -- 172.21.15.16:0/574319861 shutdown_connections 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 --2- 172.21.15.16:0/574319861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc109e30 0x7fd5fc110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 --2- 172.21.15.16:0/574319861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc1098f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 --2- 172.21.15.16:0/574319861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc102090 0x7fd5fc1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 -- 172.21.15.16:0/574319861 >> 172.21.15.16:0/574319861 conn(0x7fd5fc079e30 msgr2=0x7fd5fc07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 -- 172.21.15.16:0/574319861 shutdown_connections 2024-09-21T01:58:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.410+0000 7fd6028c3640 1 -- 172.21.15.16:0/574319861 wait complete. 2024-09-21T01:58:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.411+0000 7fd6028c3640 1 Processor -- start 2024-09-21T01:58:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.411+0000 7fd6028c3640 1 -- start start 2024-09-21T01:58:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.411+0000 7fd6028c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc102090 0x7fd5fc19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd6028c3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:16.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd6028c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd6028c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc113880 con 0x7fd5fc109e30 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd6028c3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc113700 con 0x7fd5fc102090 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd6028c3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc113a00 con 0x7fd5fc1094e0 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50888/0 (socket says 172.21.15.16:50888) 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 -- 172.21.15.16:0/1795049565 learned_addr learned my addr 172.21.15.16:0/1795049565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd5fb7fe640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd5fbfff640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc102090 0x7fd5fc19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 -- 172.21.15.16:0/1795049565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 msgr2=0x7fd5fc19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 -- 172.21.15.16:0/1795049565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc102090 msgr2=0x7fd5fc19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc102090 0x7fd5fc19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 -- 172.21.15.16:0/1795049565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc1a0320 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd5fb7fe640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd600e60640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eb10 tx=0x7fd5f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.412+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4004b20 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.413+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd5f4004cc0 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.413+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc1a0610 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.413+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4010690 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.413+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc1a6d70 con 0x7fd5fc109e30 2024-09-21T01:58:16.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.414+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc1024a0 con 0x7fd5fc109e30 2024-09-21T01:58:16.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.415+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fd5f4005020 con 0x7fd5fc109e30 2024-09-21T01:58:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.416+0000 7fd5f97fa640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 0x7fd5e40b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.416+0000 7fd5fbfff640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 0x7fd5e40b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.416+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd5f40e5cb0 con 0x7fd5fc109e30 2024-09-21T01:58:16.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.416+0000 7fd5fbfff640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 0x7fd5e40b8a60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009bd0 tx=0x7fd5ec005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:16.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.420+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd5f40ae0a0 con 0x7fd5fc109e30 2024-09-21T01:58:16.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.610+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7fd5fc10bc20 con 0x7fd5fc109e30 2024-09-21T01:58:16.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.611+0000 7fd5f97fa640 1 -- 172.21.15.16:0/1795049565 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v28)=0 v28) ==== 108+0+0 (secure 0 0 0) 0x7fd5f40059e0 con 0x7fd5fc109e30 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 msgr2=0x7fd5e40b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 0x7fd5e40b8a60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009bd0 tx=0x7fd5ec005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 msgr2=0x7fd5fc19fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400eb10 tx=0x7fd5f400c2d0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 shutdown_connections 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5e40b6550 0x7fd5e40b8a60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5fc109e30 0x7fd5fc19fbf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5fc1094e0 0x7fd5fc19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 --2- 172.21.15.16:0/1795049565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc102090 0x7fd5fc19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 >> 172.21.15.16:0/1795049565 conn(0x7fd5fc079e30 msgr2=0x7fd5fc107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 shutdown_connections 2024-09-21T01:58:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:16.613+0000 7fd6028c3640 1 -- 172.21.15.16:0/1795049565 wait complete. 2024-09-21T01:58:16.979 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:16 smithi016 ceph-mon[31245]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:58:16.979 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:16 smithi016 ceph-mon[31245]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:58:17.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:16 smithi123 ceph-mon[33453]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:58:17.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:16 smithi123 ceph-mon[33453]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:58:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:16 smithi161 ceph-mon[34061]: Reconfiguring mon.c (monmap changed)... 2024-09-21T01:58:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:16 smithi161 ceph-mon[34061]: Reconfiguring daemon mon.c on smithi161 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 -- 172.21.15.16:0/3219778741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 msgr2=0x7f69e810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 --2- 172.21.15.16:0/3219778741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e810f520 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009900 tx=0x7f69e002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 -- 172.21.15.16:0/3219778741 shutdown_connections 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 --2- 172.21.15.16:0/3219778741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e810f520 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 --2- 172.21.15.16:0/3219778741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 0x7f69e81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 --2- 172.21.15.16:0/3219778741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.467+0000 7f69f067a640 1 -- 172.21.15.16:0/3219778741 >> 172.21.15.16:0/3219778741 conn(0x7f69e8079730 msgr2=0x7f69e8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.468+0000 7f69f067a640 1 -- 172.21.15.16:0/3219778741 shutdown_connections 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.468+0000 7f69f067a640 1 -- 172.21.15.16:0/3219778741 wait complete. 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.468+0000 7f69f067a640 1 Processor -- start 2024-09-21T01:58:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.468+0000 7f69f067a640 1 -- start start 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.468+0000 7f69f067a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69f067a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 0x7f69e819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69f067a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69f067a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69e81126f0 con 0x7f69e8108cf0 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69f067a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69e8112570 con 0x7f69e81083a0 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69f067a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f69e8112870 con 0x7f69e8102a60 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46866/0 (socket says 172.21.15.16:46866) 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69ee416640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 -- 172.21.15.16:0/4080945406 learned_addr learned my addr 172.21.15.16:0/4080945406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69edc15640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 0x7f69e819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 -- 172.21.15.16:0/4080945406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 msgr2=0x7f69e819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 -- 172.21.15.16:0/4080945406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 msgr2=0x7f69e819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 0x7f69e819a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 -- 172.21.15.16:0/4080945406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e81a0260 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69ee416640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69eec17640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69e0005ab0 tx=0x7f69e0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.469+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e003f070 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.470+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e81a04f0 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.470+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f69e003bae0 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.470+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0035030 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.470+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e81a6cb0 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.471+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e8103ce0 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.472+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f69e0042070 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.473+0000 7f69e77fe640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 0x7f69d00b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.473+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f69e0106110 con 0x7f69e8108cf0 2024-09-21T01:58:17.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.473+0000 7f69ee416640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 0x7f69d00b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.474+0000 7f69ee416640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 0x7f69d00b89e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d8009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.481+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f69e00d2980 con 0x7f69e8108cf0 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.687+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f69e8102e70 con 0x7f69e8108cf0 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.691+0000 7f69e77fe640 1 -- 172.21.15.16:0/4080945406 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f69e8102e70 con 0x7f69e8108cf0 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 msgr2=0x7f69d00b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 0x7f69d00b89e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d8009500 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 msgr2=0x7f69e819fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69e0005ab0 tx=0x7f69e0003c10 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 shutdown_connections 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f69d00b64d0 0x7f69d00b89e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69e8108cf0 0x7f69e819fb30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69e81083a0 0x7f69e819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 --2- 172.21.15.16:0/4080945406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69e8102a60 0x7f69e819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 >> 172.21.15.16:0/4080945406 conn(0x7f69e8079730 msgr2=0x7f69e806c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:17.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 shutdown_connections 2024-09-21T01:58:17.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.694+0000 7f69f067a640 1 -- 172.21.15.16:0/4080945406 wait complete. 2024-09-21T01:58:17.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.368+0000 7fb2aa790640 1 -- 172.21.15.16:0/2074456229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a41083a0 msgr2=0x7fb2a41087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.368+0000 7fb2aa790640 1 --2- 172.21.15.16:0/2074456229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a41083a0 0x7fb2a41087b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb298009900 tx=0x7fb29802a410 comp rx=0 tx=0).stop 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.368+0000 7fb2aa790640 1 -- 172.21.15.16:0/2074456229 shutdown_connections 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.368+0000 7fb2aa790640 1 --2- 172.21.15.16:0/2074456229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 0x7fb2a410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 --2- 172.21.15.16:0/2074456229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a41083a0 0x7fb2a41087b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 --2- 172.21.15.16:0/2074456229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a4102a60 0x7fb2a4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 -- 172.21.15.16:0/2074456229 >> 172.21.15.16:0/2074456229 conn(0x7fb2a4079730 msgr2=0x7fb2a4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 -- 172.21.15.16:0/2074456229 shutdown_connections 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 -- 172.21.15.16:0/2074456229 wait complete. 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 Processor -- start 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.369+0000 7fb2aa790640 1 -- start start 2024-09-21T01:58:17.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a41083a0 0x7fb2a419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 0x7fb2a41a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a4115a00 con 0x7fb2a4102a60 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a4115880 con 0x7fb2a41083a0 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2aa790640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a4115b80 con 0x7fb2a4108cf0 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46840/0 (socket says 172.21.15.16:46840) 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 -- 172.21.15.16:0/1132637893 learned_addr learned my addr 172.21.15.16:0/1132637893 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a8d2d640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 0x7fb2a41a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a37fe640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a41083a0 0x7fb2a419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 -- 172.21.15.16:0/1132637893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 msgr2=0x7fb2a41a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 0x7fb2a41a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 -- 172.21.15.16:0/1132637893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a41083a0 msgr2=0x7fb2a419cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a41083a0 0x7fb2a419cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 -- 172.21.15.16:0/1132637893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a41a24a0 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.370+0000 7fb2a3fff640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb29400d950 tx=0x7fb29400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.371+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294017070 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.371+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb294009c30 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.371+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a41a2790 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.371+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29400f620 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.371+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a41a8ef0 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.372+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a4102e70 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.374+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fb294005030 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.374+0000 7fb2a17fa640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 0x7fb28c0b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.374+0000 7fb2a37fe640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 0x7fb28c0b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.374+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb2940e52f0 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.375+0000 7fb2a37fe640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 0x7fb28c0b8a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb29802aa00 tx=0x7fb298000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.377+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb2940ad800 con 0x7fb2a4102a60 2024-09-21T01:58:17.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.592+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fb2a402e640 con 0x7fb2a4102a60 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.597+0000 7fb2a17fa640 1 -- 172.21.15.16:0/1132637893 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v29)=0 v29) ==== 119+0+0 (secure 0 0 0) 0x7fb294009de0 con 0x7fb2a4102a60 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.599+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 msgr2=0x7fb28c0b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.599+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 0x7fb28c0b8a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb29802aa00 tx=0x7fb298000bc0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.599+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 msgr2=0x7fb2a419c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.599+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb29400d950 tx=0x7fb29400de10 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 shutdown_connections 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb28c0b6530 0x7fb28c0b8a40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2a4108cf0 0x7fb2a41a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2a41083a0 0x7fb2a419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 --2- 172.21.15.16:0/1132637893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a4102a60 0x7fb2a419c530 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 >> 172.21.15.16:0/1132637893 conn(0x7fb2a4079730 msgr2=0x7fb2a41053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 shutdown_connections 2024-09-21T01:58:17.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:17.600+0000 7fb2aa790640 1 -- 172.21.15.16:0/1132637893 wait complete. 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1132637893' entity='client.admin' 2024-09-21T01:58:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4080945406' entity='client.admin' 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.008 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:18.008 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1132637893' entity='client.admin' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4080945406' entity='client.admin' 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.010 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.010 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:18.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1132637893' entity='client.admin' 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4080945406' entity='client.admin' 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:18.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.190+0000 7fd000205640 1 -- 172.21.15.16:0/2036648134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 msgr2=0x7fcff810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.190+0000 7fd000205640 1 --2- 172.21.15.16:0/2036648134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff810d8d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009900 tx=0x7fcfe802a410 comp rx=0 tx=0).stop 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 -- 172.21.15.16:0/2036648134 shutdown_connections 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 --2- 172.21.15.16:0/2036648134 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 0x7fcff8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 --2- 172.21.15.16:0/2036648134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff810d8d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 --2- 172.21.15.16:0/2036648134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 0x7fcff8104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 -- 172.21.15.16:0/2036648134 >> 172.21.15.16:0/2036648134 conn(0x7fcff80797b0 msgr2=0x7fcff8079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 -- 172.21.15.16:0/2036648134 shutdown_connections 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 -- 172.21.15.16:0/2036648134 wait complete. 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 Processor -- start 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.191+0000 7fd000205640 1 -- start start 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 0x7fcff819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 0x7fcff819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8113880 con 0x7fcff8105380 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8113700 con 0x7fcff8104a30 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fd000205640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8113a00 con 0x7fcff810de60 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46882/0 (socket says 172.21.15.16:46882) 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 -- 172.21.15.16:0/4199084898 learned_addr learned my addr 172.21.15.16:0/4199084898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:19.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffe7a2640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 0x7fcff819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffdfa1640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 0x7fcff819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 -- 172.21.15.16:0/4199084898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 msgr2=0x7fcff819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 0x7fcff819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 -- 172.21.15.16:0/4199084898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 msgr2=0x7fcff819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 0x7fcff819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 -- 172.21.15.16:0/4199084898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff81a0280 con 0x7fcff8105380 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.192+0000 7fcffd7a0640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8005ab0 tx=0x7fcfe8003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.193+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe803f070 con 0x7fcff8105380 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.193+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fcfe803bce0 con 0x7fcff8105380 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.193+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8035030 con 0x7fcff8105380 2024-09-21T01:58:19.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.193+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff81a0510 con 0x7fcff8105380 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.193+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff81a6cd0 con 0x7fcff8105380 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.194+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff806bed0 con 0x7fcff8105380 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.196+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fcfe8042070 con 0x7fcff8105380 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.197+0000 7fcfeeffd640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 0x7fcfe40b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.197+0000 7fcffdfa1640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 0x7fcfe40b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.197+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcfe81061b0 con 0x7fcff8105380 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.197+0000 7fcffdfa1640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 0x7fcfe40b89e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009bd0 tx=0x7fcff4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.200+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcfe80d2a20 con 0x7fcff8105380 2024-09-21T01:58:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.391+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fcff8108f40 con 0x7fcff8105380 2024-09-21T01:58:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.396+0000 7fcfeeffd640 1 -- 172.21.15.16:0/4199084898 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7fcfe80d7960 con 0x7fcff8105380 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 msgr2=0x7fcfe40b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 0x7fcfe40b89e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009bd0 tx=0x7fcff4005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 msgr2=0x7fcff819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8005ab0 tx=0x7fcfe8003c80 comp rx=0 tx=0).stop 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 shutdown_connections 2024-09-21T01:58:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcfe40b64d0 0x7fcfe40b89e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcff810de60 0x7fcff819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcff8105380 0x7fcff819a8b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 --2- 172.21.15.16:0/4199084898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcff8104a30 0x7fcff819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 >> 172.21.15.16:0/4199084898 conn(0x7fcff80797b0 msgr2=0x7fcff81009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 shutdown_connections 2024-09-21T01:58:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.398+0000 7fd000205640 1 -- 172.21.15.16:0/4199084898 wait complete. 2024-09-21T01:58:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 -- 172.21.15.16:0/754609742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c107360 msgr2=0x7f094c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 --2- 172.21.15.16:0/754609742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c107360 0x7f094c107770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0938009900 tx=0x7f093802a410 comp rx=0 tx=0).stop 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 -- 172.21.15.16:0/754609742 shutdown_connections 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 --2- 172.21.15.16:0/754609742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c1106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 --2- 172.21.15.16:0/754609742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c107360 0x7f094c107770 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 --2- 172.21.15.16:0/754609742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c102160 0x7f094c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.517+0000 7f09530f7640 1 -- 172.21.15.16:0/754609742 >> 172.21.15.16:0/754609742 conn(0x7f094c079ea0 msgr2=0x7f094c07a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- 172.21.15.16:0/754609742 shutdown_connections 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- 172.21.15.16:0/754609742 wait complete. 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 Processor -- start 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- start start 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c107360 0x7f094c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f094c1138b0 con 0x7f094c102160 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f094c113730 con 0x7f094c107cb0 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.518+0000 7f09530f7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f094c113a30 con 0x7f094c107360 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46896/0 (socket says 172.21.15.16:46896) 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 -- 172.21.15.16:0/1458930987 learned_addr learned my addr 172.21.15.16:0/1458930987 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f094bfff640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c107360 0x7f094c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0951694640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 -- 172.21.15.16:0/1458930987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c107360 msgr2=0x7f094c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c107360 0x7f094c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 -- 172.21.15.16:0/1458930987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 msgr2=0x7f094c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 -- 172.21.15.16:0/1458930987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094c1a03b0 con 0x7f094c102160 2024-09-21T01:58:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0951694640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0950e93640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f094000c890 tx=0x7f094000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940004220 con 0x7f094c102160 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f0940007e20 con 0x7f094c102160 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094c1a06a0 con 0x7f094c102160 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.519+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09400047a0 con 0x7f094c102160 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.520+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f094c1a6e00 con 0x7f094c102160 2024-09-21T01:58:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.521+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094c102570 con 0x7f094c102160 2024-09-21T01:58:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.523+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f09400079b0 con 0x7f094c102160 2024-09-21T01:58:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.524+0000 7f0949ffb640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 0x7f093c0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.524+0000 7f094bfff640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 0x7f093c0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.524+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f09400e57d0 con 0x7f094c102160 2024-09-21T01:58:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.524+0000 7f094bfff640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 0x7f093c0b8a60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0938005940 tx=0x7f09380058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.527+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0940014030 con 0x7f094c102160 2024-09-21T01:58:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.717+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f094c10bc60 con 0x7f094c102160 2024-09-21T01:58:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.722+0000 7f0949ffb640 1 -- 172.21.15.16:0/1458930987 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v32)=0 v32) ==== 129+0+0 (secure 0 0 0) 0x7f09400adce0 con 0x7f094c102160 2024-09-21T01:58:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.724+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 msgr2=0x7f093c0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.724+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 0x7f093c0b8a60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0938005940 tx=0x7f09380058d0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 msgr2=0x7f094c19a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f094000c890 tx=0x7f094000cd50 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 shutdown_connections 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f093c0b6550 0x7f093c0b8a60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f094c107cb0 0x7f094c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f094c107360 0x7f094c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 --2- 172.21.15.16:0/1458930987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f094c102160 0x7f094c19a440 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 >> 172.21.15.16:0/1458930987 conn(0x7f094c079ea0 msgr2=0x7f094c106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 shutdown_connections 2024-09-21T01:58:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.725+0000 7f09530f7640 1 -- 172.21.15.16:0/1458930987 wait complete. 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 -- 172.21.15.16:0/3301229387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102f60 msgr2=0x7fda1410fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 --2- 172.21.15.16:0/3301229387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102f60 0x7fda1410fc80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda0802a480 comp rx=0 tx=0).stop 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 -- 172.21.15.16:0/3301229387 shutdown_connections 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 --2- 172.21.15.16:0/3301229387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda141101c0 0x7fda141125e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 --2- 172.21.15.16:0/3301229387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102f60 0x7fda1410fc80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 --2- 172.21.15.16:0/3301229387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda14102610 0x7fda14102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 -- 172.21.15.16:0/3301229387 >> 172.21.15.16:0/3301229387 conn(0x7fda14079650 msgr2=0x7fda14079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 -- 172.21.15.16:0/3301229387 shutdown_connections 2024-09-21T01:58:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.944+0000 7fda1bfd0640 1 -- 172.21.15.16:0/3301229387 wait complete. 2024-09-21T01:58:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 Processor -- start 2024-09-21T01:58:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 -- start start 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda14102f60 0x7fda1419cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 0x7fda141a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda14115a90 con 0x7fda14102610 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda14115910 con 0x7fda141101c0 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1bfd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda14115c10 con 0x7fda14102f60 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46902/0 (socket says 172.21.15.16:46902) 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 -- 172.21.15.16:0/251508013 learned_addr learned my addr 172.21.15.16:0/251508013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1956b640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda14102f60 0x7fda1419cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1a56d640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 0x7fda141a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 -- 172.21.15.16:0/251508013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda14102f60 msgr2=0x7fda1419cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda14102f60 0x7fda1419cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 -- 172.21.15.16:0/251508013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 msgr2=0x7fda141a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 0x7fda141a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda19d6c640 1 -- 172.21.15.16:0/251508013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda141a24f0 con 0x7fda14102610 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.945+0000 7fda1a56d640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 0x7fda141a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda19d6c640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda0400eab0 tx=0x7fda0400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0400ce40 con 0x7fda14102610 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fda040118e0 con 0x7fda14102610 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda1bfd0640 1 -- 172.21.15.16:0/251508013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda141a27e0 con 0x7fda14102610 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04010830 con 0x7fda14102610 2024-09-21T01:58:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.946+0000 7fda1bfd0640 1 -- 172.21.15.16:0/251508013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda141a8f40 con 0x7fda14102610 2024-09-21T01:58:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.947+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f4005290 con 0x7fda14102610 2024-09-21T01:58:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.949+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fda04011470 con 0x7fda14102610 2024-09-21T01:58:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.949+0000 7fda12ffd640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 0x7fd9fc0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.950+0000 7fda1956b640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 0x7fd9fc0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.950+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fda040e5d20 con 0x7fda14102610 2024-09-21T01:58:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.950+0000 7fda1956b640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 0x7fd9fc0b8a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda08005ab0 tx=0x7fda08005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:19.953+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fda04016030 con 0x7fda14102610 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4199084898' entity='client.admin' 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4199084898' entity='client.admin' 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:20.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 -- 172.21.15.16:0/2214491857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed58105300 msgr2=0x7fed5810d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 --2- 172.21.15.16:0/2214491857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed58105300 0x7fed5810d850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fed48009ca0 tx=0x7fed4802a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 -- 172.21.15.16:0/2214491857 shutdown_connections 2024-09-21T01:58:20.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 --2- 172.21.15.16:0/2214491857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed5810de10 0x7fed58110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 --2- 172.21.15.16:0/2214491857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed58105300 0x7fed5810d850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 --2- 172.21.15.16:0/2214491857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed58104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 -- 172.21.15.16:0/2214491857 >> 172.21.15.16:0/2214491857 conn(0x7fed58079730 msgr2=0x7fed58079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 -- 172.21.15.16:0/2214491857 shutdown_connections 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.129+0000 7fed5ec42640 1 -- 172.21.15.16:0/2214491857 wait complete. 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 Processor -- start 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 -- start start 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed5819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58105300 0x7fed5819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed58103100 con 0x7fed5810de10 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed58102f80 con 0x7fed58105300 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5ec42640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed58103280 con 0x7fed581049b0 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5d1df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5d1df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46914/0 (socket says 172.21.15.16:46914) 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5d1df640 1 -- 172.21.15.16:0/3903105640 learned_addr learned my addr 172.21.15.16:0/3903105640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed5c9de640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed5819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.130+0000 7fed57fff640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58105300 0x7fed5819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 -- 172.21.15.16:0/3903105640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 msgr2=0x7fed5819c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed5819c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 -- 172.21.15.16:0/3903105640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58105300 msgr2=0x7fed5819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58105300 0x7fed5819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 -- 172.21.15.16:0/3903105640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed581a24f0 con 0x7fed5810de10 2024-09-21T01:58:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5c9de640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed5819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5d1df640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fed5000d950 tx=0x7fed5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50017030 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fed50004020 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500039c0 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed581a27e0 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.131+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed581a8f40 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.133+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed58104dc0 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.134+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fed5001a070 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.135+0000 7fed55ffb640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 0x7fed440b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.135+0000 7fed5c9de640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 0x7fed440b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.135+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fed500e4e00 con 0x7fed5810de10 2024-09-21T01:58:20.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.136+0000 7fed5c9de640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 0x7fed440b8a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5806bc60 tx=0x7fed40005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.139+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fed50021070 con 0x7fed5810de10 2024-09-21T01:58:20.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.146+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fd9f4005580 con 0x7fda14102610 2024-09-21T01:58:20.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.150+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v33)=0 v33) ==== 123+0+0 (secure 0 0 0) 0x7fda040ae230 con 0x7fda14102610 2024-09-21T01:58:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.151+0000 7fda12ffd640 1 -- 172.21.15.16:0/251508013 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda040ad640 con 0x7fda14102610 2024-09-21T01:58:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.151+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed500ac6a0 con 0x7fed5810de10 2024-09-21T01:58:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.152+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 msgr2=0x7fd9fc0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.152+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 0x7fd9fc0b8a60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda08005ab0 tx=0x7fda08005a20 comp rx=0 tx=0).stop 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.152+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 msgr2=0x7fda1419c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.152+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda0400eab0 tx=0x7fda0400ef70 comp rx=0 tx=0).stop 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 shutdown_connections 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd9fc0b6550 0x7fd9fc0b8a60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda141101c0 0x7fda141a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda14102f60 0x7fda1419cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 --2- 172.21.15.16:0/251508013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14102610 0x7fda1419c580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 >> 172.21.15.16:0/251508013 conn(0x7fda14079650 msgr2=0x7fda14111720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 shutdown_connections 2024-09-21T01:58:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.153+0000 7fda10ff9640 1 -- 172.21.15.16:0/251508013 wait complete. 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4199084898' entity='client.admin' 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.336+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fed58107980 con 0x7fed5810de10 2024-09-21T01:58:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.340+0000 7fed55ffb640 1 -- 172.21.15.16:0/3903105640 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v34)=0 v34) ==== 121+0+0 (secure 0 0 0) 0x7fed500ad290 con 0x7fed5810de10 2024-09-21T01:58:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 msgr2=0x7fed440b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 0x7fed440b8a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5806bc60 tx=0x7fed40005f90 comp rx=0 tx=0).stop 2024-09-21T01:58:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 msgr2=0x7fed581a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fed5000d950 tx=0x7fed5000de10 comp rx=0 tx=0).stop 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 shutdown_connections 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fed440b6550 0x7fed440b8a60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed5810de10 0x7fed581a1dc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed58105300 0x7fed5819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 --2- 172.21.15.16:0/3903105640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed581049b0 0x7fed5819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 >> 172.21.15.16:0/3903105640 conn(0x7fed58079730 msgr2=0x7fed5810f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 shutdown_connections 2024-09-21T01:58:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.343+0000 7fed5ec42640 1 -- 172.21.15.16:0/3903105640 wait complete. 2024-09-21T01:58:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 -- 172.21.15.16:0/2015311147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d4108cf0 msgr2=0x7f34d410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2015311147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d4108cf0 0x7f34d410f520 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009ca0 tx=0x7f34cc02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 -- 172.21.15.16:0/2015311147 shutdown_connections 2024-09-21T01:58:20.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2015311147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d4108cf0 0x7f34d410f520 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2015311147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d41083a0 0x7f34d41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2015311147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 -- 172.21.15.16:0/2015311147 >> 172.21.15.16:0/2015311147 conn(0x7f34d4079730 msgr2=0x7f34d4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 -- 172.21.15.16:0/2015311147 shutdown_connections 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.852+0000 7f34da3c7640 1 -- 172.21.15.16:0/2015311147 wait complete. 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 Processor -- start 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 -- start start 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d4108cf0 0x7f34d419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4113960 con 0x7f34d41083a0 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34d41137e0 con 0x7f34d4102a60 2024-09-21T01:58:20.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34da3c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4113ae0 con 0x7f34d4108cf0 2024-09-21T01:58:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46940/0 (socket says 172.21.15.16:46940) 2024-09-21T01:58:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34cbfff640 1 -- 172.21.15.16:0/2712142392 learned_addr learned my addr 172.21.15.16:0/2712142392 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34d8964640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d4108cf0 0x7f34d419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.853+0000 7f34d3fff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 -- 172.21.15.16:0/2712142392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d4108cf0 msgr2=0x7f34d419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d4108cf0 0x7f34d419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 -- 172.21.15.16:0/2712142392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 msgr2=0x7f34d419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 -- 172.21.15.16:0/2712142392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d41a02c0 con 0x7f34d41083a0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34d3fff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34cbfff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00e9f0 tx=0x7f34bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00ccc0 con 0x7f34d41083a0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34bc0054b0 con 0x7f34d41083a0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc010430 con 0x7f34d41083a0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d41a05b0 con 0x7f34d41083a0 2024-09-21T01:58:20.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.854+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d41a6d10 con 0x7f34d41083a0 2024-09-21T01:58:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.855+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d4102e70 con 0x7f34d41083a0 2024-09-21T01:58:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.857+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f34bc0046c0 con 0x7f34d41083a0 2024-09-21T01:58:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.858+0000 7f34d1ffb640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 0x7f34c00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.858+0000 7f34d3fff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 0x7f34c00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:20.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.858+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f34bc0e5cf0 con 0x7f34d41083a0 2024-09-21T01:58:20.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.858+0000 7f34d3fff640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 0x7f34c00b8a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34c40058c0 tx=0x7f34c4005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:20.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:20.861+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f34bc056710 con 0x7f34d41083a0 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1458930987' entity='client.admin' 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/251508013' entity='client.admin' 2024-09-21T01:58:21.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3903105640' entity='client.admin' 2024-09-21T01:58:21.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.052+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f34d419b480 con 0x7f34d41083a0 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1458930987' entity='client.admin' 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:21.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/251508013' entity='client.admin' 2024-09-21T01:58:21.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3903105640' entity='client.admin' 2024-09-21T01:58:21.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.056+0000 7f34d1ffb640 1 -- 172.21.15.16:0/2712142392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v35)=0 v35) ==== 111+0+0 (secure 0 0 0) 0x7f34d419b480 con 0x7f34d41083a0 2024-09-21T01:58:21.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 msgr2=0x7f34c00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 0x7f34c00b8a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34c40058c0 tx=0x7f34c4005850 comp rx=0 tx=0).stop 2024-09-21T01:58:21.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 msgr2=0x7f34d419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00e9f0 tx=0x7f34bc00eeb0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 shutdown_connections 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f34c00b6550 0x7f34c00b8a60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34d4108cf0 0x7f34d419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d41083a0 0x7f34d419a8f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 --2- 172.21.15.16:0/2712142392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34d4102a60 0x7f34d419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.058+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 >> 172.21.15.16:0/2712142392 conn(0x7f34d4079730 msgr2=0x7f34d41043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.059+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 shutdown_connections 2024-09-21T01:58:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.059+0000 7f34da3c7640 1 -- 172.21.15.16:0/2712142392 wait complete. 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1458930987' entity='client.admin' 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/251508013' entity='client.admin' 2024-09-21T01:58:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3903105640' entity='client.admin' 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 -- 172.21.15.16:0/1922487457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 msgr2=0x7f1a88103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 --2- 172.21.15.16:0/1922487457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a88103f70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009900 tx=0x7f1a8402a410 comp rx=0 tx=0).stop 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 -- 172.21.15.16:0/1922487457 shutdown_connections 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 --2- 172.21.15.16:0/1922487457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a88109df0 0x7f1a88110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 --2- 172.21.15.16:0/1922487457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a881094a0 0x7f1a881098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 --2- 172.21.15.16:0/1922487457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a88103f70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 -- 172.21.15.16:0/1922487457 >> 172.21.15.16:0/1922487457 conn(0x7f1a8807b850 msgr2=0x7f1a8807bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 -- 172.21.15.16:0/1922487457 shutdown_connections 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 -- 172.21.15.16:0/1922487457 wait complete. 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.617+0000 7f1a90437640 1 Processor -- start 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 -- start start 2024-09-21T01:58:21.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 0x7f1a8819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a88109df0 0x7f1a8819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1a881139f0 con 0x7f1a88103b60 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88113870 con 0x7f1a881094a0 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a90437640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88113b70 con 0x7f1a88109df0 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46958/0 (socket says 172.21.15.16:46958) 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 -- 172.21.15.16:0/1827904682 learned_addr learned my addr 172.21.15.16:0/1827904682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8d9d2640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 0x7f1a8819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 -- 172.21.15.16:0/1827904682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a88109df0 msgr2=0x7f1a8819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a88109df0 0x7f1a8819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 -- 172.21.15.16:0/1827904682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 msgr2=0x7f1a8819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 0x7f1a8819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8e1d3640 1 -- 172.21.15.16:0/1827904682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a881a0370 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.618+0000 7f1a8d9d2640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 0x7f1a8819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a8e1d3640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a8402a930 tx=0x7f1a84003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8403f070 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a84003e40 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a881a0600 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84035030 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.619+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a881a6dc0 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.620+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a88103f70 con 0x7f1a88103b60 2024-09-21T01:58:21.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.622+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f1a84042070 con 0x7f1a88103b60 2024-09-21T01:58:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.622+0000 7f1a7f7fe640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 0x7f1a740b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.622+0000 7f1a8d9d2640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 0x7f1a740b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.622+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1a84106400 con 0x7f1a88103b60 2024-09-21T01:58:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.623+0000 7f1a8d9d2640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 0x7f1a740b89e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a881062a0 tx=0x7f1a78008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:21.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.626+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1a840ce890 con 0x7f1a88103b60 2024-09-21T01:58:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.819+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f1a8802e640 con 0x7f1a88103b60 2024-09-21T01:58:21.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.824+0000 7f1a7f7fe640 1 -- 172.21.15.16:0/1827904682 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v36)=0 v36) ==== 121+0+0 (secure 0 0 0) 0x7f1a840d2c70 con 0x7f1a88103b60 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 msgr2=0x7f1a740b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 0x7f1a740b89e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a881062a0 tx=0x7f1a78008040 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 msgr2=0x7f1a8819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a8402a930 tx=0x7f1a84003c20 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 shutdown_connections 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1a740b64d0 0x7f1a740b89e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a88109df0 0x7f1a8819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a881094a0 0x7f1a8819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 --2- 172.21.15.16:0/1827904682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a88103b60 0x7f1a8819a400 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 >> 172.21.15.16:0/1827904682 conn(0x7f1a8807b850 msgr2=0x7f1a88107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 shutdown_connections 2024-09-21T01:58:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:21.826+0000 7f1a90437640 1 -- 172.21.15.16:0/1827904682 wait complete. 2024-09-21T01:58:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2712142392' entity='client.admin' 2024-09-21T01:58:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:22.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2712142392' entity='client.admin' 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2712142392' entity='client.admin' 2024-09-21T01:58:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.215 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.216 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi016') 2024-09-21T01:58:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:22 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1827904682' entity='client.admin' 2024-09-21T01:58:22.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:22 smithi016 ceph-mon[31245]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:23.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:22 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1827904682' entity='client.admin' 2024-09-21T01:58:23.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:22 smithi161 ceph-mon[34061]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:22 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1827904682' entity='client.admin' 2024-09-21T01:58:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:22 smithi123 ceph-mon[33453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 -- 172.21.15.16:0/2751747311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 msgr2=0x7f8b501066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2751747311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b501066f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8b40009900 tx=0x7f8b4002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 -- 172.21.15.16:0/2751747311 shutdown_connections 2024-09-21T01:58:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2751747311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50106c30 0x7f8b50110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2751747311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b501066f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2751747311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50102000 0x7f8b50102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 -- 172.21.15.16:0/2751747311 >> 172.21.15.16:0/2751747311 conn(0x7f8b50079e40 msgr2=0x7f8b5007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 -- 172.21.15.16:0/2751747311 shutdown_connections 2024-09-21T01:58:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.577+0000 7f8b57df2640 1 -- 172.21.15.16:0/2751747311 wait complete. 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 Processor -- start 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 -- start start 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 0x7f8b5019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50106c30 0x7f8b5019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50113980 con 0x7f8b501062e0 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50113800 con 0x7f8b50106c30 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b57df2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50113b00 con 0x7f8b50102000 2024-09-21T01:58:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b5538d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b5538d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46968/0 (socket says 172.21.15.16:46968) 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b5538d640 1 -- 172.21.15.16:0/2182403150 learned_addr learned my addr 172.21.15.16:0/2182403150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b5638f640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50106c30 0x7f8b5019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b55b8e640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 0x7f8b5019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.578+0000 7f8b5538d640 1 -- 172.21.15.16:0/2182403150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 msgr2=0x7f8b5019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b5538d640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 0x7f8b5019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b5538d640 1 -- 172.21.15.16:0/2182403150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50106c30 msgr2=0x7f8b5019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b5538d640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50106c30 0x7f8b5019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b5538d640 1 -- 172.21.15.16:0/2182403150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b501a0360 con 0x7f8b501062e0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b55b8e640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 0x7f8b5019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b5538d640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005940 tx=0x7f8b40004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4003f030 con 0x7f8b501062e0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b501a05f0 con 0x7f8b501062e0 2024-09-21T01:58:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b40004440 con 0x7f8b501062e0 2024-09-21T01:58:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40035070 con 0x7f8b501062e0 2024-09-21T01:58:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.579+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b501a6db0 con 0x7f8b501062e0 2024-09-21T01:58:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.580+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b50102410 con 0x7f8b501062e0 2024-09-21T01:58:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.582+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f8b40042030 con 0x7f8b501062e0 2024-09-21T01:58:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.583+0000 7f8b46ffd640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 0x7f8b3c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.583+0000 7f8b55b8e640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 0x7f8b3c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.583+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8b40105f90 con 0x7f8b501062e0 2024-09-21T01:58:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.584+0000 7f8b55b8e640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 0x7f8b3c0b89e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0058c0 tx=0x7f8b4c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.587+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8b400ce420 con 0x7f8b501062e0 2024-09-21T01:58:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.778+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f8b5010bc40 con 0x7f8b501062e0 2024-09-21T01:58:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.778+0000 7f8b46ffd640 1 -- 172.21.15.16:0/2182403150 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v36)=0 v36) ==== 106+0+0 (secure 0 0 0) 0x7f8b400d2800 con 0x7f8b501062e0 2024-09-21T01:58:23.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 msgr2=0x7f8b3c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:23.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 0x7f8b3c0b89e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0058c0 tx=0x7f8b4c005850 comp rx=0 tx=0).stop 2024-09-21T01:58:23.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 msgr2=0x7f8b5019a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:23.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005940 tx=0x7f8b40004040 comp rx=0 tx=0).stop 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 shutdown_connections 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b3c0b64d0 0x7f8b3c0b89e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b50106c30 0x7f8b5019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b501062e0 0x7f8b5019a990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 --2- 172.21.15.16:0/2182403150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b50102000 0x7f8b5019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 >> 172.21.15.16:0/2182403150 conn(0x7f8b50079e40 msgr2=0x7f8b50107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 shutdown_connections 2024-09-21T01:58:23.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:23.780+0000 7f8b57df2640 1 -- 172.21.15.16:0/2182403150 wait complete. 2024-09-21T01:58:23.977 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:58:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:24 smithi016 ceph-mon[31245]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:24 smithi123 ceph-mon[33453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:24 smithi161 ceph-mon[34061]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 -- 172.21.15.16:0/2605136331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c105270 msgr2=0x7fa95c10d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 --2- 172.21.15.16:0/2605136331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c105270 0x7fa95c10d8b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa950009900 tx=0x7fa95002a410 comp rx=0 tx=0).stop 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 -- 172.21.15.16:0/2605136331 shutdown_connections 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 --2- 172.21.15.16:0/2605136331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c10ddf0 0x7fa95c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 --2- 172.21.15.16:0/2605136331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c105270 0x7fa95c10d8b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 --2- 172.21.15.16:0/2605136331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c104840 0x7fa95c104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 -- 172.21.15.16:0/2605136331 >> 172.21.15.16:0/2605136331 conn(0x7fa95c079730 msgr2=0x7fa95c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 -- 172.21.15.16:0/2605136331 shutdown_connections 2024-09-21T01:58:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.721+0000 7fa961409640 1 -- 172.21.15.16:0/2605136331 wait complete. 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 Processor -- start 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 -- start start 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c105270 0x7fa95c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c10ddf0 0x7fa95c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c113970 con 0x7fa95c104840 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c1137f0 con 0x7fa95c10ddf0 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.722+0000 7fa961409640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c113af0 con 0x7fa95c105270 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46988/0 (socket says 172.21.15.16:46988) 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 -- 172.21.15.16:0/458723552 learned_addr learned my addr 172.21.15.16:0/458723552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95a7fc640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c105270 0x7fa95c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95b7fe640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c10ddf0 0x7fa95c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 -- 172.21.15.16:0/458723552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c105270 msgr2=0x7fa95c19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c105270 0x7fa95c19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 -- 172.21.15.16:0/458723552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c10ddf0 msgr2=0x7fa95c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c10ddf0 0x7fa95c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 -- 172.21.15.16:0/458723552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c1a0330 con 0x7fa95c104840 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa95affd640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00ea80 tx=0x7fa94c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:25.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00ce40 con 0x7fa95c104840 2024-09-21T01:58:25.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95c1a0620 con 0x7fa95c104840 2024-09-21T01:58:25.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa94c011900 con 0x7fa95c104840 2024-09-21T01:58:25.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.723+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c01a9e0 con 0x7fa95c104840 2024-09-21T01:58:25.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.724+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95c1a6d80 con 0x7fa95c104840 2024-09-21T01:58:25.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.725+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c104c50 con 0x7fa95c104840 2024-09-21T01:58:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.727+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fa94c01ab80 con 0x7fa95c104840 2024-09-21T01:58:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.728+0000 7fa943fff640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 0x7fa9440b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.728+0000 7fa95a7fc640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 0x7fa9440b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.728+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa94c0e5dd0 con 0x7fa95c104840 2024-09-21T01:58:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.729+0000 7fa95a7fc640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 0x7fa9440b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa95002a930 tx=0x7fa950005a40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:25.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.731+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa94c058710 con 0x7fa95c104840 2024-09-21T01:58:25.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.923+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7fa95c19b4f0 con 0x7fa95c104840 2024-09-21T01:58:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.927+0000 7fa943fff640 1 -- 172.21.15.16:0/458723552 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v37)=0 v37) ==== 115+0+0 (secure 0 0 0) 0x7fa94c016030 con 0x7fa95c104840 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 msgr2=0x7fa9440b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 0x7fa9440b89e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa95002a930 tx=0x7fa950005a40 comp rx=0 tx=0).stop 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 msgr2=0x7fa95c19a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00ea80 tx=0x7fa94c00ef40 comp rx=0 tx=0).stop 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 shutdown_connections 2024-09-21T01:58:25.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9440b64d0 0x7fa9440b89e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa95c10ddf0 0x7fa95c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa95c105270 0x7fa95c19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 --2- 172.21.15.16:0/458723552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa95c104840 0x7fa95c19a3c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 >> 172.21.15.16:0/458723552 conn(0x7fa95c079730 msgr2=0x7fa95c0ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 shutdown_connections 2024-09-21T01:58:25.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:25.930+0000 7fa961409640 1 -- 172.21.15.16:0/458723552 wait complete. 2024-09-21T01:58:26.829 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.829 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.829 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.830 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.830 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.830 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.830 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.830 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.831 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8372ca74-77bc-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi016') 2024-09-21T01:58:26.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config dump 2024-09-21T01:58:27.090 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:27.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/458723552' entity='client.admin' 2024-09-21T01:58:27.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:27.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:27.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:27.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:27.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/458723552' entity='client.admin' 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/458723552' entity='client.admin' 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 -- 172.21.15.16:0/4025964379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc108d20 msgr2=0x7ff6cc10f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 --2- 172.21.15.16:0/4025964379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc108d20 0x7ff6cc10f550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009ca0 tx=0x7ff6c402a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 -- 172.21.15.16:0/4025964379 shutdown_connections 2024-09-21T01:58:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 --2- 172.21.15.16:0/4025964379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc108d20 0x7ff6cc10f550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 --2- 172.21.15.16:0/4025964379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc1083d0 0x7ff6cc1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 --2- 172.21.15.16:0/4025964379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 -- 172.21.15.16:0/4025964379 >> 172.21.15.16:0/4025964379 conn(0x7ff6cc079790 msgr2=0x7ff6cc079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 -- 172.21.15.16:0/4025964379 shutdown_connections 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.544+0000 7ff6d3053640 1 -- 172.21.15.16:0/4025964379 wait complete. 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 Processor -- start 2024-09-21T01:58:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 -- start start 2024-09-21T01:58:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc1083d0 0x7ff6cc1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 0x7ff6cc19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1128e0 con 0x7ff6cc102ae0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc112760 con 0x7ff6cc108d20 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d3053640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc112a60 con 0x7ff6cc1083d0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54236/0 (socket says 172.21.15.16:54236) 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 -- 172.21.15.16:0/3988932680 learned_addr learned my addr 172.21.15.16:0/3988932680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d15f0640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 0x7ff6cc19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6cbfff640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc1083d0 0x7ff6cc1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 -- 172.21.15.16:0/3988932680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc1083d0 msgr2=0x7ff6cc1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc1083d0 0x7ff6cc1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 -- 172.21.15.16:0/3988932680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 msgr2=0x7ff6cc19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 0x7ff6cc19ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d0def640 1 -- 172.21.15.16:0/3988932680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc19f100 con 0x7ff6cc102ae0 2024-09-21T01:58:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.545+0000 7ff6d15f0640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 0x7ff6cc19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6d0def640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000d950 tx=0x7ff6c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0017070 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6c0009c30 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc19f3f0 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c000f620 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.546+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc1a5c10 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.547+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6cc103d10 con 0x7ff6cc102ae0 2024-09-21T01:58:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.549+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7ff6c0010460 con 0x7ff6cc102ae0 2024-09-21T01:58:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.549+0000 7ff6c9ffb640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 0x7ff6bc0b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.550+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff6c00e5390 con 0x7ff6cc102ae0 2024-09-21T01:58:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.550+0000 7ff6cbfff640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 0x7ff6bc0b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:27.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.550+0000 7ff6cbfff640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 0x7ff6bc0b8a90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8005e10 tx=0x7ff6b8005da0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:27.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.554+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6c00ad820 con 0x7ff6cc102ae0 2024-09-21T01:58:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.745+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7ff6cc102ef0 con 0x7ff6cc102ae0 2024-09-21T01:58:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.746+0000 7ff6c9ffb640 1 -- 172.21.15.16:0/3988932680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+21924 (secure 0 0 0) 0x7ff6c00b1c00 con 0x7ff6cc102ae0 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global dev auth_debug true 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global basic container_image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 * 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global dev debug_asserts_on_shutdown true 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global basic log_to_file true 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global basic log_to_journald false 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global basic log_to_stderr false 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_allow_pool_delete true 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_clock_drift_allowed 1.000000 2024-09-21T01:58:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_cluster_log_to_file true 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_max_pg_per_osd 10000 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced mon_warn_on_too_few_osds false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global dev ms_die_on_bug true 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global dev ms_die_on_old_message true 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced osd_pool_default_erasure_code_profile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2024-09-21T01:58:27.748 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced auth_service_ticket_ttl 240.000000 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced debug_mon 20/20 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced debug_ms 1/1 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced debug_paxos 20/20 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_data_avail_warn 5 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_down_mkfs_grace 300 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_mgr_mkfs_grace 240 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon dev mon_osd_prime_pg_temp true 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_osd_reporter_subtree_level osd 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mon advanced public_network 172.21.0.0/20 * 2024-09-21T01:58:27.749 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced cephsqlite_lock_renewal_timeout 900000 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced client_mount_timeout 30 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced debug_cephsqlite 20/20 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced debug_client 20/20 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced debug_mgr 20/20 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced debug_ms 1/1 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/cephadm/container_init True * 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/cephadm/migration_current 7 * 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/snap_schedule/allow_m_granularity true 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/snap_schedule/dump_on_update true 2024-09-21T01:58:27.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mgr/snap_schedule/log_level debug 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev bdev_debug_aio true 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev bluestore_block_size 96636764160 * 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced bluestore_compression_mode aggressive 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev bluestore_fsck_on_mount true 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced debug_bluefs 20/20 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced debug_bluestore 20/20 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced debug_ms 1/1 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced debug_osd 20/20 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced debug_rocksdb 10/10 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_misdirected_ops true 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_op_order true 2024-09-21T01:58:27.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_pg_log_writeout true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_shutdown true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_verify_cached_snaps true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_verify_missing_on_start true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd dev osd_debug_verify_stray_on_activate true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_failsafe_full_ratio 0.950000 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_max_omap_entries_per_request 10000 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_mclock_profile high_recovery_ops 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_memory_target_autotune true 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_objectstore bluestore * 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_op_queue debug_random * 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_op_queue_cut_off debug_random * 2024-09-21T01:58:27.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_recover_clone_overlap true 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_recovery_max_chunk 1048576 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_scrub_load_threshold 5.000000 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_scrub_max_interval 600.000000 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd advanced osd_shutdown_pgref_assert true 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced debug_mds 20/20 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced debug_ms 2/2 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_bal_merge_size 5 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_bal_replicate_threshold 1.000000 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_bal_split_bits 3 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_bal_split_size 100 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds dev mds_debug_frag true 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds dev mds_debug_scatterstat true 2024-09-21T01:58:27.753 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:mds dev mds_verify_scatter true 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client advanced client_mount_timeout 600 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client advanced debug_client 20/20 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client advanced debug_ms 1/1 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client.rgw advanced rgw_cache_enabled true 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client.rgw advanced rgw_enable_ops_log true 2024-09-21T01:58:27.754 INFO:teuthology.orchestra.run.smithi016.stdout:client.rgw advanced rgw_enable_usage_log true 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 msgr2=0x7ff6bc0b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 0x7ff6bc0b8a90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8005e10 tx=0x7ff6b8005da0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 msgr2=0x7ff6cc199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000d950 tx=0x7ff6c000de10 comp rx=0 tx=0).stop 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 shutdown_connections 2024-09-21T01:58:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff6bc0b6580 0x7ff6bc0b8a90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6cc108d20 0x7ff6cc19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6cc1083d0 0x7ff6cc1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 --2- 172.21.15.16:0/3988932680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6cc102ae0 0x7ff6cc199250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 >> 172.21.15.16:0/3988932680 conn(0x7ff6cc079790 msgr2=0x7ff6cc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 shutdown_connections 2024-09-21T01:58:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:27.748+0000 7ff6d3053640 1 -- 172.21.15.16:0/3988932680 wait complete. 2024-09-21T01:58:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:27 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3988932680' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T01:58:28.339 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:27 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3988932680' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T01:58:28.368 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T01:58:28.368 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T01:58:28.368 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T01:58:28.385 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T01:58:28.385 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-21T01:58:28.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:27 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3988932680' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-21 01:57:04.515705415 +0000 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-21 01:52:51.510206533 +0000 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-21 01:52:51.510206533 +0000 2024-09-21T01:58:28.442 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-21 01:52:51.510206533 +0000 2024-09-21T01:58:28.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T01:58:28.507 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-21T01:58:28.507 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-21T01:58:28.508 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.3019e-05 s, 5.5 MB/s 2024-09-21T01:58:28.509 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T01:58:28.565 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-21T01:58:28.621 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T01:58:28.621 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:28.621 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T01:58:28.621 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:28.621 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:28.622 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-21 01:57:04.550704837 +0000 2024-09-21T01:58:28.622 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-21 01:52:51.827201316 +0000 2024-09-21T01:58:28.622 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-21 01:52:51.827201316 +0000 2024-09-21T01:58:28.622 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-21 01:52:51.827201316 +0000 2024-09-21T01:58:28.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T01:58:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-21T01:58:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-21T01:58:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000142781 s, 3.6 MB/s 2024-09-21T01:58:28.687 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T01:58:28.744 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-21 01:57:04.585704258 +0000 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-21 01:52:52.156195903 +0000 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-21 01:52:52.156195903 +0000 2024-09-21T01:58:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-21 01:52:52.156195903 +0000 2024-09-21T01:58:28.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T01:58:28.865 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-21T01:58:28.865 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-21T01:58:28.865 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000109624 s, 4.7 MB/s 2024-09-21T01:58:28.866 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T01:58:28.923 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-21 01:57:04.624703614 +0000 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-21 01:52:52.479190588 +0000 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-21 01:52:52.479190588 +0000 2024-09-21T01:58:28.979 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-21 01:52:52.479190588 +0000 2024-09-21T01:58:28.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T01:58:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-21T01:58:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-21T01:58:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000110781 s, 4.6 MB/s 2024-09-21T01:58:29.045 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T01:58:29.102 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-21T01:58:29.103 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T01:58:29.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T01:58:29.122 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-09-21T01:58:29.180 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-21 01:58:07.552937302 +0000 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-21 01:52:44.891368104 +0000 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-21 01:52:44.891368104 +0000 2024-09-21T01:58:29.181 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-09-21 01:52:44.891368104 +0000 2024-09-21T01:58:29.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T01:58:29.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:28 smithi123 ceph-mon[33453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:29.250 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-21T01:58:29.250 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-21T01:58:29.250 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000223737 s, 2.3 MB/s 2024-09-21T01:58:29.252 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T01:58:29.310 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-21 01:58:07.552937302 +0000 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-21 01:52:45.217360249 +0000 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-21 01:52:45.217360249 +0000 2024-09-21T01:58:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-09-21 01:52:45.217360249 +0000 2024-09-21T01:58:29.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T01:58:29.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:28 smithi016 ceph-mon[31245]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:29.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:28 smithi161 ceph-mon[34061]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:29.433 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-21T01:58:29.433 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-21T01:58:29.434 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000211454 s, 2.4 MB/s 2024-09-21T01:58:29.435 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T01:58:29.492 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-09-21T01:58:29.549 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-21 01:58:07.552937302 +0000 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-21 01:52:45.544352370 +0000 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-21 01:52:45.544352370 +0000 2024-09-21T01:58:29.550 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-09-21 01:52:45.544352370 +0000 2024-09-21T01:58:29.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T01:58:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-21T01:58:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-21T01:58:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000259694 s, 2.0 MB/s 2024-09-21T01:58:29.616 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T01:58:29.674 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-09-21T01:58:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T01:58:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:29.730 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-21T01:58:29.730 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:29.730 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:29.731 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-21 01:58:07.553937279 +0000 2024-09-21T01:58:29.731 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-21 01:52:45.867344587 +0000 2024-09-21T01:58:29.731 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-21 01:52:45.867344587 +0000 2024-09-21T01:58:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-09-21 01:52:45.867344587 +0000 2024-09-21T01:58:29.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T01:58:29.794 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-21T01:58:29.794 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-21T01:58:29.794 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00021262 s, 2.4 MB/s 2024-09-21T01:58:29.795 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T01:58:29.852 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-21T01:58:29.853 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T01:58:29.872 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T01:58:29.872 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-09-21T01:58:29.931 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-21 01:58:05.257653808 +0000 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-21 01:52:44.896821987 +0000 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-21 01:52:44.896821987 +0000 2024-09-21T01:58:29.932 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-21 01:52:44.896821987 +0000 2024-09-21T01:58:29.933 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T01:58:30.002 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-21T01:58:30.002 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-21T01:58:30.002 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000269848 s, 1.9 MB/s 2024-09-21T01:58:30.004 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T01:58:30.063 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-09-21T01:58:30.120 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T01:58:30.120 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:30.120 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-21 01:58:05.257653808 +0000 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-21 01:52:45.232814610 +0000 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-21 01:52:45.232814610 +0000 2024-09-21T01:58:30.121 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-21 01:52:45.232814610 +0000 2024-09-21T01:58:30.121 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T01:58:30.189 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-21T01:58:30.190 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-21T01:58:30.190 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000197772 s, 2.6 MB/s 2024-09-21T01:58:30.191 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T01:58:30.267 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-09-21T01:58:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T01:58:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:30.333 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T01:58:30.333 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:30.333 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:30.334 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-21 01:58:05.258653786 +0000 2024-09-21T01:58:30.334 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-21 01:52:45.561807387 +0000 2024-09-21T01:58:30.334 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-21 01:52:45.561807387 +0000 2024-09-21T01:58:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-21 01:52:45.561807387 +0000 2024-09-21T01:58:30.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T01:58:30.408 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-21T01:58:30.408 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-21T01:58:30.408 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00025699 s, 2.0 MB/s 2024-09-21T01:58:30.410 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T01:58:30.474 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-09-21T01:58:30.535 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-21 01:58:05.258653786 +0000 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-21 01:52:45.879800405 +0000 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-21 01:52:45.879800405 +0000 2024-09-21T01:58:30.536 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-21 01:52:45.879800405 +0000 2024-09-21T01:58:30.536 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T01:58:30.609 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-21T01:58:30.609 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-21T01:58:30.609 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000245756 s, 2.1 MB/s 2024-09-21T01:58:30.610 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T01:58:30.672 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-09-21T01:58:30.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T01:58:30.935 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:31.259 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:30 smithi016 ceph-mon[31245]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:31.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:30 smithi123 ceph-mon[33453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:31.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:30 smithi161 ceph-mon[34061]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:33.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:58:33.247 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:32 smithi016 ceph-mon[31245]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:33.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-09-21T01:58:33.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:32 smithi123 ceph-mon[33453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:33.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:32 smithi161 ceph-mon[34061]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:58:33.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 -- 172.21.15.16:0/3373914949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e810ddf0 msgr2=0x7f58e8110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/3373914949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e810ddf0 0x7f58e8110300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009ca0 tx=0x7f58e002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 -- 172.21.15.16:0/3373914949 shutdown_connections 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/3373914949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e810ddf0 0x7f58e8110300 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/3373914949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e8105270 0x7f58e810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/3373914949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e8104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 -- 172.21.15.16:0/3373914949 >> 172.21.15.16:0/3373914949 conn(0x7f58e8079730 msgr2=0x7f58e8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 -- 172.21.15.16:0/3373914949 shutdown_connections 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.892+0000 7f58eeb0e640 1 -- 172.21.15.16:0/3373914949 wait complete. 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 Processor -- start 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 -- start start 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:33.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 0x7f58e819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58e8113970 con 0x7f58e8105270 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58e81137f0 con 0x7f58e810ddf0 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58eeb0e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f58e8113af0 con 0x7f58e8104840 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58e7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.893+0000 7f58ec8aa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54248/0 (socket says 172.21.15.16:54248) 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 -- 172.21.15.16:0/113691061 learned_addr learned my addr 172.21.15.16:0/113691061 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58ed0ab640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 0x7f58e819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 -- 172.21.15.16:0/113691061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 msgr2=0x7f58e819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 -- 172.21.15.16:0/113691061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 msgr2=0x7f58e819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 0x7f58e819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:33.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 -- 172.21.15.16:0/113691061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e81a02f0 con 0x7f58e8105270 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58ec8aa640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58ed0ab640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 0x7f58e819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e7fff640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00c970 tx=0x7f58dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc007bb0 con 0x7f58e8105270 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e81a05e0 con 0x7f58e8105270 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58dc007d50 con 0x7f58e8105270 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.894+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc004da0 con 0x7f58e8105270 2024-09-21T01:58:33.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.895+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e81a6d40 con 0x7f58e8105270 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.895+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e8104c50 con 0x7f58e8105270 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.897+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f58dc004f80 con 0x7f58e8105270 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.898+0000 7f58e5ffb640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 0x7f58d00b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.898+0000 7f58ec8aa640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 0x7f58d00b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.898+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f58dc0e5430 con 0x7f58e8105270 2024-09-21T01:58:33.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.898+0000 7f58ec8aa640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 0x7f58d00b89e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58d8005fd0 tx=0x7f58d800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:58:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:33.902+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f58dc014030 con 0x7f58e8105270 2024-09-21T01:58:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:34.099+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f58e8101d90 con 0x7f58d00b64d0 2024-09-21T01:58:35.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:34 smithi016 ceph-mon[31245]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:35.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:34 smithi016 ceph-mon[31245]: from='client.14445 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:35.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:58:35.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:58:35.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:34 smithi123 ceph-mon[33453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:34 smithi123 ceph-mon[33453]: from='client.14445 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:58:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:58:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:35.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:34 smithi161 ceph-mon[34061]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:35.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:34 smithi161 ceph-mon[34061]: from='client.14445 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:35.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:58:35.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:58:35.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:36 smithi123 ceph-mon[33453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"} : dispatch 2024-09-21T01:58:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"}]': finished 2024-09-21T01:58:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:36 smithi123 ceph-mon[33453]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T01:58:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:36 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:36 smithi016 ceph-mon[31245]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:36 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"} : dispatch 2024-09-21T01:58:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:36 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"}]': finished 2024-09-21T01:58:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:36 smithi016 ceph-mon[31245]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T01:58:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:36 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:36 smithi161 ceph-mon[34061]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:36 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"} : dispatch 2024-09-21T01:58:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:36 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/520097352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f8f1230-9fd6-4eb8-8804-c04765d2734c"}]': finished 2024-09-21T01:58:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:36 smithi161 ceph-mon[34061]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T01:58:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:36 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:38 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/971469579' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:58:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:38 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/971469579' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:58:38.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:38 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/971469579' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:58:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:39 smithi123 ceph-mon[33453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:39.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:39 smithi016 ceph-mon[31245]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:39.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:39 smithi161 ceph-mon[34061]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:41.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:41 smithi123 ceph-mon[33453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:41.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:41 smithi016 ceph-mon[31245]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:41.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:41 smithi161 ceph-mon[34061]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:43 smithi016 ceph-mon[31245]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:43 smithi123 ceph-mon[33453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:43.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:43 smithi161 ceph-mon[34061]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:45 smithi016 ceph-mon[31245]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:45 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T01:58:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:45 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:45.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:45 smithi123 ceph-mon[33453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:45.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:45 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T01:58:45.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:45 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:45.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:45 smithi161 ceph-mon[34061]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:45.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:45 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T01:58:45.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:45 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:46.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:46 smithi123 ceph-mon[33453]: Deploying daemon osd.0 on smithi016 2024-09-21T01:58:46.337 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:46 smithi016 ceph-mon[31245]: Deploying daemon osd.0 on smithi016 2024-09-21T01:58:46.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:46 smithi161 ceph-mon[34061]: Deploying daemon osd.0 on smithi016 2024-09-21T01:58:47.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:47 smithi123 ceph-mon[33453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:47.337 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:47 smithi016 ceph-mon[31245]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:47.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:47 smithi161 ceph-mon[34061]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:49.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:49 smithi016 ceph-mon[31245]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:49.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:49 smithi161 ceph-mon[34061]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:49 smithi123 ceph-mon[33453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:51.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:51 smithi016 ceph-mon[31245]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:51.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:51 smithi161 ceph-mon[34061]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:51.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:51 smithi123 ceph-mon[33453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:52.036 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:52.036 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:52.036 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:52.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:52.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:52.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:52.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:52.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:53 smithi016 ceph-mon[31245]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:53.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:53 smithi161 ceph-mon[34061]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:53 smithi123 ceph-mon[33453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:54.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:54.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:54.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:54.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:54.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:58:55.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:58:55.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:55.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.731+0000 7f58e5ffb640 1 -- 172.21.15.16:0/113691061 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f58e8101d90 con 0x7f58d00b64d0 2024-09-21T01:58:55.732 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-09-21T01:58:55.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 msgr2=0x7f58d00b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:55.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 0x7f58d00b89e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58d8005fd0 tx=0x7f58d800b410 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 msgr2=0x7f58e819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00c970 tx=0x7f58dc00ce30 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 shutdown_connections 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f58d00b64d0 0x7f58d00b89e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e810ddf0 0x7f58e819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58e8105270 0x7f58e819a920 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 --2- 172.21.15.16:0/113691061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58e8104840 0x7f58e819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.733+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 >> 172.21.15.16:0/113691061 conn(0x7f58e8079730 msgr2=0x7f58e8100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.734+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 shutdown_connections 2024-09-21T01:58:55.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:58:55.734+0000 7f58eeb0e640 1 -- 172.21.15.16:0/113691061 wait complete. 2024-09-21T01:58:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:55 smithi123 ceph-mon[33453]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:55 smithi016 ceph-mon[31245]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:56.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:56.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.030 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:55 smithi161 ceph-mon[34061]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:58:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:58:56.376 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.0.service 2024-09-21T01:58:56.379 INFO:tasks.cephadm:Deploying osd.1 on smithi123 with /dev/vg_nvme/lv_4... 2024-09-21T01:58:56.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T01:58:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:58:57.471 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:58:57 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T01:58:57.182+0000 7f8f8916a640 -1 osd.0 0 log_to_monitors true 2024-09-21T01:58:58.015 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:57 smithi123 ceph-mon[33453]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:58.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:57 smithi123 ceph-mon[33453]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T01:58:58.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:57 smithi016 ceph-mon[31245]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:58.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:57 smithi016 ceph-mon[31245]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T01:58:58.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:57 smithi161 ceph-mon[34061]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:58:58.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:57 smithi161 ceph-mon[34061]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T01:58:58.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T01:58:58.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T01:58:58.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T01:58:58.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:58.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T01:58:58.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T01:58:58.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:58 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:59.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T01:58:59.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T01:58:59.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T01:58:59.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:59.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T01:58:59.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T01:58:59.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:58 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T01:58:59.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:58 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:58:59.383 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T01:58:59.408 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T01:58:59.190+0000 7f8f85901640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T01:58:59.408 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T01:58:59.195+0000 7f8f7cee2640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:58:59.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-09-21T01:58:59.660 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T01:59:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' 2024-09-21T01:59:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:58:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 -- 172.21.15.123:0/3575280966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 msgr2=0x7f514c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:00.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 --2- 172.21.15.123:0/3575280966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c1087b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f513c009ca0 tx=0x7f513c02a8c0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 -- 172.21.15.123:0/3575280966 shutdown_connections 2024-09-21T01:59:00.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 --2- 172.21.15.123:0/3575280966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c108cf0 0x7f514c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 --2- 172.21.15.123:0/3575280966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c1087b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.135+0000 7f5152d5d640 1 --2- 172.21.15.123:0/3575280966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 0x7f514c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.136+0000 7f5152d5d640 1 -- 172.21.15.123:0/3575280966 >> 172.21.15.123:0/3575280966 conn(0x7f514c079730 msgr2=0x7f514c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.136+0000 7f5152d5d640 1 -- 172.21.15.123:0/3575280966 shutdown_connections 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.136+0000 7f5152d5d640 1 -- 172.21.15.123:0/3575280966 wait complete. 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.136+0000 7f5152d5d640 1 Processor -- start 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 -- start start 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 0x7f514c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1083a0 0x7f514c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f514c112860 con 0x7f514c102a60 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1126e0 con 0x7f514c108cf0 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5152d5d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1129e0 con 0x7f514c1083a0 2024-09-21T01:59:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49522/0 (socket says 172.21.15.123:49522) 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 -- 172.21.15.123:0/4215429622 learned_addr learned my addr 172.21.15.123:0/4215429622 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f514bfff640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1083a0 0x7f514c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f5150af9640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 0x7f514c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 -- 172.21.15.123:0/4215429622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1083a0 msgr2=0x7f514c199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1083a0 0x7f514c199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 -- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 msgr2=0x7f514c199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 0x7f514c199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.137+0000 7f51512fa640 1 -- 172.21.15.123:0/4215429622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c19f2e0 con 0x7f514c108cf0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f51512fa640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f513400d950 tx=0x7f513400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134017030 con 0x7f514c108cf0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5134004020 con 0x7f514c108cf0 2024-09-21T01:59:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134003ab0 con 0x7f514c108cf0 2024-09-21T01:59:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c19f5d0 con 0x7f514c108cf0 2024-09-21T01:59:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.138+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1a5d30 con 0x7f514c108cf0 2024-09-21T01:59:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.139+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c102e70 con 0x7f514c108cf0 2024-09-21T01:59:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.141+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f513401a070 con 0x7f514c108cf0 2024-09-21T01:59:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.142+0000 7f5149ffb640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 0x7f51380b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.142+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1702+0+0 (secure 0 0 0) 0x7f51340e4f80 con 0x7f514c108cf0 2024-09-21T01:59:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.142+0000 7f5150af9640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 0x7f51380b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.143+0000 7f5150af9640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 0x7f51380b8a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5144009b60 tx=0x7f5144005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:00.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.147+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5134021070 con 0x7f514c108cf0 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' 2024-09-21T01:59:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:58:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060]' entity='osd.0' 2024-09-21T01:59:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:58:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:00.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:00.335+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f514c063800 con 0x7f51380b6550 2024-09-21T01:59:01.011 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T01:59:01.011 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T01:59:01.011 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi016 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] boot 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:01.012 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T01:59:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T01:59:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi016 2024-09-21T01:59:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-21T01:59:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] boot 2024-09-21T01:59:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T01:59:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:01.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T01:59:01.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi016 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi016 to 17092M 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: osd.0 [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] boot 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:01.914 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:01 smithi123 ceph-mon[33453]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:01.914 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:01 smithi123 ceph-mon[33453]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:01 smithi016 ceph-mon[31245]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:01 smithi016 ceph-mon[31245]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:01 smithi161 ceph-mon[34061]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:59:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:01 smithi161 ceph-mon[34061]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:03.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T01:59:03.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/2886748535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"}]': finished 2024-09-21T01:59:03.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T01:59:03.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/2886748535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"}]': finished 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T01:59:03.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/2886748535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"} : dispatch 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354cac3a-106d-4c37-a1f3-dcbd1bdf142d"}]': finished 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T01:59:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:04.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:03 smithi123 ceph-mon[33453]: pgmap v33: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:04.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/2648106446' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:03 smithi016 ceph-mon[31245]: pgmap v33: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/2648106446' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:03 smithi161 ceph-mon[34061]: pgmap v33: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/2648106446' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:04 smithi016 ceph-mon[31245]: pgmap v35: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:05.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:04 smithi161 ceph-mon[34061]: pgmap v35: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:05.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:04 smithi123 ceph-mon[33453]: pgmap v35: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:06 smithi123 ceph-mon[33453]: pgmap v36: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:06 smithi016 ceph-mon[31245]: pgmap v36: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:07.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:06 smithi161 ceph-mon[34061]: pgmap v36: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:09.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:08 smithi123 ceph-mon[33453]: pgmap v37: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:08 smithi016 ceph-mon[31245]: pgmap v37: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:09.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:08 smithi161 ceph-mon[34061]: pgmap v37: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:11.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:10 smithi123 ceph-mon[33453]: pgmap v38: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:11.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T01:59:11.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:10 smithi016 ceph-mon[31245]: pgmap v38: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T01:59:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:11.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:10 smithi161 ceph-mon[34061]: pgmap v38: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:11.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T01:59:11.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:12 smithi123 ceph-mon[33453]: Deploying daemon osd.1 on smithi123 2024-09-21T01:59:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:12 smithi016 ceph-mon[31245]: Deploying daemon osd.1 on smithi123 2024-09-21T01:59:12.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:12 smithi161 ceph-mon[34061]: Deploying daemon osd.1 on smithi123 2024-09-21T01:59:13.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:13 smithi123 ceph-mon[33453]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:13 smithi016 ceph-mon[31245]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:13.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:13 smithi161 ceph-mon[34061]: pgmap v39: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:15.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:15 smithi123 ceph-mon[33453]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:15 smithi016 ceph-mon[31245]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:15.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:15 smithi161 ceph-mon[34061]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:17.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:17 smithi123 ceph-mon[33453]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:17 smithi016 ceph-mon[31245]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:17.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:17 smithi161 ceph-mon[34061]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:18 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:18 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:18 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:18 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:18 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:18 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:18 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:18 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:18 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:19.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:19 smithi123 ceph-mon[33453]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:19 smithi016 ceph-mon[31245]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:19 smithi161 ceph-mon[34061]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:20.979 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:20.980 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:20.980 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:20.980 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:20.980 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.549+0000 7f5149ffb640 1 -- 172.21.15.123:0/4215429622 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f514c063800 con 0x7f51380b6550 2024-09-21T01:59:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 1 on host 'smithi123' 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 msgr2=0x7f51380b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 0x7f51380b8a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5144009b60 tx=0x7f5144005f90 comp rx=0 tx=0).stop 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 msgr2=0x7f514c19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f513400d950 tx=0x7f513400de10 comp rx=0 tx=0).stop 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 shutdown_connections 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51380b6550 0x7f51380b8a60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:21.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.551+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c108cf0 0x7f514c19ebb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.552+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1083a0 0x7f514c199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.552+0000 7f5152d5d640 1 --2- 172.21.15.123:0/4215429622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c102a60 0x7f514c199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.552+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 >> 172.21.15.123:0/4215429622 conn(0x7f514c079730 msgr2=0x7f514c104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:59:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.552+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 shutdown_connections 2024-09-21T01:59:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T01:59:21.552+0000 7f5152d5d640 1 -- 172.21.15.123:0/4215429622 wait complete. 2024-09-21T01:59:21.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:21 smithi123 ceph-mon[33453]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:21.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:21.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:21.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:22.004 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:21 smithi016 ceph-mon[31245]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:22.004 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:22.004 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:22.004 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:21 smithi161 ceph-mon[34061]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:22.195 DEBUG:teuthology.orchestra.run.smithi123:osd.1> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.1.service 2024-09-21T01:59:22.197 INFO:tasks.cephadm:Deploying osd.2 on smithi161 with /dev/vg_nvme/lv_4... 2024-09-21T01:59:22.197 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T01:59:22.450 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:59:23.271 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T01:59:22.971+0000 7f86ffa98640 -1 osd.1 0 log_to_monitors true 2024-09-21T01:59:23.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:23 smithi161 ceph-mon[34061]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:23.859 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:23 smithi161 ceph-mon[34061]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:23.859 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:23 smithi161 ceph-mon[34061]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:24.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:23 smithi123 ceph-mon[33453]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:24.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:23 smithi123 ceph-mon[33453]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:24.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:23 smithi123 ceph-mon[33453]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:23 smithi016 ceph-mon[31245]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:23 smithi016 ceph-mon[31245]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:23 smithi016 ceph-mon[31245]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:24.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:24 smithi123 ceph-mon[33453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T01:59:24.928 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T01:59:24.956 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi161:vg_nvme/lv_4 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:24 smithi016 ceph-mon[31245]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T01:59:25.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T01:59:25.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T01:59:25.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: from='osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:25.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:25.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T01:59:25.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:24 smithi161 ceph-mon[34061]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T01:59:25.204 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T01:59:25.520 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T01:59:25.169+0000 7f86fba1a640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T01:59:25.520 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T01:59:25.173+0000 7f86f3810640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:59:25.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 -- 172.21.15.161:0/3067861737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 msgr2=0x7fa4b0110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:25.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 --2- 172.21.15.161:0/3067861737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b0110650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009900 tx=0x7fa4a802a410 comp rx=0 tx=0).stop 2024-09-21T01:59:25.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 -- 172.21.15.161:0/3067861737 shutdown_connections 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 --2- 172.21.15.161:0/3067861737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b0110650 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 --2- 172.21.15.161:0/3067861737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b01094d0 0x7fa4b01098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 --2- 172.21.15.161:0/3067861737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b0103b90 0x7fa4b0103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 -- 172.21.15.161:0/3067861737 >> 172.21.15.161:0/3067861737 conn(0x7fa4b00ff5e0 msgr2=0x7fa4b0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 -- 172.21.15.161:0/3067861737 shutdown_connections 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.672+0000 7fa4b515c640 1 -- 172.21.15.161:0/3067861737 wait complete. 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 Processor -- start 2024-09-21T01:59:25.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 -- start start 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b0103b90 0x7fa4b019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b01094d0 0x7fa4b019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b0113990 con 0x7fa4b01094d0 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b0113810 con 0x7fa4b0103b90 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4b515c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b0113b10 con 0x7fa4b0109e20 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4af59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4af59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37842/0 (socket says 172.21.15.161:37842) 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4af59e640 1 -- 172.21.15.161:0/1459528829 learned_addr learned my addr 172.21.15.161:0/1459528829 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4aed9d640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b0103b90 0x7fa4b019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.673+0000 7fa4ae59c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b01094d0 0x7fa4b019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 -- 172.21.15.161:0/1459528829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b0103b90 msgr2=0x7fa4b019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:25.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b0103b90 0x7fa4b019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 -- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b01094d0 msgr2=0x7fa4b019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b01094d0 0x7fa4b019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 -- 172.21.15.161:0/1459528829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b01a02d0 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4af59e640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8005ab0 tx=0x7fa4a80040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a803f030 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b01a0560 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4a8003a80 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8035070 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.674+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b00755f0 con 0x7fa4b0109e20 2024-09-21T01:59:25.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.675+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b0104e10 con 0x7fa4b0109e20 2024-09-21T01:59:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.677+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7fa4a8042030 con 0x7fa4b0109e20 2024-09-21T01:59:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.677+0000 7fa497fff640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 0x7fa4980b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.677+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2244+0+0 (secure 0 0 0) 0x7fa4a81064b0 con 0x7fa4b0109e20 2024-09-21T01:59:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.677+0000 7fa4aed9d640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 0x7fa4980b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.678+0000 7fa4aed9d640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 0x7fa4980b89e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0005fd0 tx=0x7fa4a0009870 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:25.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.681+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa4a80ce5d0 con 0x7fa4b0109e20 2024-09-21T01:59:25.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:25.868+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa4b0063800 con 0x7fa4980b64d0 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:25.870 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:25.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:25.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='osd.1 ' entity='osd.1' 2024-09-21T01:59:25.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:26.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T01:59:26.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='osd.1 ' entity='osd.1' 2024-09-21T01:59:26.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='osd.1 ' entity='osd.1' 2024-09-21T01:59:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi123 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi123 to 17088M 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] boot 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T01:59:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:27.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T01:59:27.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T01:59:27.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi123 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi123 to 17088M 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] boot 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T01:59:27.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi123 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi123 to 17088M 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:27.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: osd.1 [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] boot 2024-09-21T01:59:27.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T01:59:27.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:59:28.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:27 smithi123 ceph-mon[33453]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:28.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:27 smithi123 ceph-mon[33453]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:27 smithi016 ceph-mon[31245]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:27 smithi016 ceph-mon[31245]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:28.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:27 smithi161 ceph-mon[34061]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:28.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:27 smithi161 ceph-mon[34061]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:59:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T01:59:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/2655572477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"}]': finished 2024-09-21T01:59:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T01:59:29.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:29.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/2491081603' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/2655572477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"}]': finished 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/2491081603' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/2655572477' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"} : dispatch 2024-09-21T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21a8f0b2-d4d2-4fdd-9302-7475215892e4"}]': finished 2024-09-21T01:59:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T01:59:29.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:29.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/2491081603' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:29 smithi016 ceph-mon[31245]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:30.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:29 smithi161 ceph-mon[34061]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:30.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:29 smithi123 ceph-mon[33453]: pgmap v51: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:30 smithi016 ceph-mon[31245]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:31.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:30 smithi161 ceph-mon[34061]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:31.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:30 smithi123 ceph-mon[33453]: pgmap v53: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:33.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:32 smithi123 ceph-mon[33453]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:33.402 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:32 smithi161 ceph-mon[34061]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:33.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:32 smithi016 ceph-mon[31245]: pgmap v54: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:35.157 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:35 smithi161 ceph-mon[34061]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:35 smithi123 ceph-mon[33453]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:35 smithi016 ceph-mon[31245]: pgmap v55: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:37 smithi123 ceph-mon[33453]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T01:59:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:37 smithi123 ceph-mon[33453]: Deploying daemon osd.2 on smithi161 2024-09-21T01:59:37.354 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:37 smithi161 ceph-mon[34061]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:37.354 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T01:59:37.355 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:37.355 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:37 smithi161 ceph-mon[34061]: Deploying daemon osd.2 on smithi161 2024-09-21T01:59:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:37 smithi016 ceph-mon[31245]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T01:59:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:37 smithi016 ceph-mon[31245]: Deploying daemon osd.2 on smithi161 2024-09-21T01:59:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:39 smithi161 ceph-mon[34061]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:39 smithi123 ceph-mon[33453]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:39.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:39 smithi016 ceph-mon[31245]: pgmap v57: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:41.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:41 smithi123 ceph-mon[33453]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:41.287 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:41 smithi161 ceph-mon[34061]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:41.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:41 smithi016 ceph-mon[31245]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:43 smithi161 ceph-mon[34061]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:43 smithi123 ceph-mon[33453]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:43 smithi016 ceph-mon[31245]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:45.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:45 smithi161 ceph-mon[34061]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:45.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:45 smithi123 ceph-mon[33453]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:45 smithi016 ceph-mon[31245]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:46.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.339+0000 7fa497fff640 1 -- 172.21.15.161:0/1459528829 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa4b0063800 con 0x7fa4980b64d0 2024-09-21T01:59:46.340 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 2 on host 'smithi161' 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.342+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 msgr2=0x7fa4980b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.342+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 0x7fa4980b89e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0005fd0 tx=0x7fa4a0009870 comp rx=0 tx=0).stop 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.342+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 msgr2=0x7fa4b019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.342+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8005ab0 tx=0x7fa4a80040e0 comp rx=0 tx=0).stop 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 shutdown_connections 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa4980b64d0 0x7fa4980b89e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:46.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4b0109e20 0x7fa4b019fba0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4b01094d0 0x7fa4b019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 --2- 172.21.15.161:0/1459528829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b0103b90 0x7fa4b019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 >> 172.21.15.161:0/1459528829 conn(0x7fa4b00ff5e0 msgr2=0x7fa4b0101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:59:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 shutdown_connections 2024-09-21T01:59:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T01:59:46.343+0000 7fa4b515c640 1 -- 172.21.15.161:0/1459528829 wait complete. 2024-09-21T01:59:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:46.985 DEBUG:teuthology.orchestra.run.smithi161:osd.2> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.2.service 2024-09-21T01:59:46.988 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-09-21T01:59:46.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T01:59:47.259 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:59:47.575 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:47 smithi016 ceph-mon[31245]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:47.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:47.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:47.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:47 smithi161 ceph-mon[34061]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:47 smithi123 ceph-mon[33453]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:59:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:48.295 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T01:59:48.277+0000 7f421b715640 -1 osd.2 0 log_to_monitors true 2024-09-21T01:59:48.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:48 smithi016 ceph-mon[31245]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:48.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:48 smithi016 ceph-mon[31245]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:48.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:48 smithi161 ceph-mon[34061]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:48.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:48 smithi161 ceph-mon[34061]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:48.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:48 smithi123 ceph-mon[33453]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:48.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:48 smithi123 ceph-mon[33453]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:59:49.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:49.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T01:59:49.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T01:59:49.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:49.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:49 smithi016 ceph-mon[31245]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-mon[34061]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T01:59:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: from='osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T01:59:49.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:49.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:49 smithi123 ceph-mon[33453]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T01:59:49.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T01:59:49.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-09-21T01:59:50.165 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T01:59:49.792+0000 7f4217eac640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T01:59:50.165 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T01:59:49.797+0000 7f420f48d640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:59:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T01:59:50.621 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T01:59:50.621 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T01:59:50.621 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.621 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi161 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:50 smithi016 ceph-mon[31245]: from='osd.2 ' entity='osd.2' 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi161 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:50 smithi161 ceph-mon[34061]: from='osd.2 ' entity='osd.2' 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 -- 172.21.15.16:0/3178680514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f201810de60 msgr2=0x7f2018110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 --2- 172.21.15.16:0/3178680514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f201810de60 0x7f2018110330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2010009ca0 tx=0x7f201002a8c0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 -- 172.21.15.16:0/3178680514 shutdown_connections 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 --2- 172.21.15.16:0/3178680514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f201810de60 0x7f2018110330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 --2- 172.21.15.16:0/3178680514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 --2- 172.21.15.16:0/3178680514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2018104a30 0x7f2018104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 -- 172.21.15.16:0/3178680514 >> 172.21.15.16:0/3178680514 conn(0x7f2018079760 msgr2=0x7f2018079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 -- 172.21.15.16:0/3178680514 shutdown_connections 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.717+0000 7f201d76f640 1 -- 172.21.15.16:0/3178680514 wait complete. 2024-09-21T01:59:50.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 Processor -- start 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 -- start start 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201810de60 0x7f201819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20181139c0 con 0x7f2018104a30 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2018113840 con 0x7f201810de60 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f201d76f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2018113b40 con 0x7f2018105380 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38942/0 (socket says 172.21.15.16:38942) 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 -- 172.21.15.16:0/2915357914 learned_addr learned my addr 172.21.15.16:0/2915357914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f20177fe640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201810de60 0x7f201819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f200e7fc640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 -- 172.21.15.16:0/2915357914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 msgr2=0x7f201819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201819a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 -- 172.21.15.16:0/2915357914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201810de60 msgr2=0x7f201819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201810de60 0x7f201819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f2016ffd640 1 -- 172.21.15.16:0/2915357914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20181a0240 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.718+0000 7f200e7fc640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f2016ffd640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f200800c970 tx=0x7f200800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008007c10 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2008007db0 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200800f7d0 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20181a0530 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.719+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20181a6d50 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.720+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201806c1f0 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.723+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 154473+0+0 (secure 0 0 0) 0x7f2008018480 con 0x7f2018104a30 2024-09-21T01:59:50.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.723+0000 7f2014ff9640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 0x7f20040b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:59:50.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.724+0000 7f200e7fc640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 0x7f20040b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:59:50.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.724+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2786+0+0 (secure 0 0 0) 0x7f20080e6330 con 0x7f2018104a30 2024-09-21T01:59:50.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.724+0000 7f200e7fc640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 0x7f20040b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2018102440 tx=0x7f2000005e50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:59:50.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.727+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f20080ae350 con 0x7f2018104a30 2024-09-21T01:59:50.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T01:59:50.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi161 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:50.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:50 smithi123 ceph-mon[33453]: from='osd.2 ' entity='osd.2' 2024-09-21T01:59:50.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:50.926+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2018063800 con 0x7f20040b6500 2024-09-21T01:59:51.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T01:59:51.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T01:59:51.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi161 to 17088M 2024-09-21T01:59:51.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:51.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] boot 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:51.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi161 to 17088M 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] boot 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T01:59:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T01:59:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi161 to 17088M 2024-09-21T01:59:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: osd.2 [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] boot 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:59:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:59:52.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:52 smithi016 ceph-mon[31245]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:52.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:52 smithi016 ceph-mon[31245]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T01:59:52.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:52 smithi161 ceph-mon[34061]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:52.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:52 smithi161 ceph-mon[34061]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T01:59:52.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:52 smithi123 ceph-mon[33453]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:52.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:52 smithi123 ceph-mon[33453]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T01:59:53.644 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: pgmap v68: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"} : dispatch 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"}]': finished 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:53.645 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: pgmap v68: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"} : dispatch 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"}]': finished 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: pgmap v68: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"} : dispatch 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3115120132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb7c5f89-293f-4a90-9a26-3ba287b3f072"}]': finished 2024-09-21T01:59:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: osdmap e21: 4 total, 3 up, 4 in 2024-09-21T01:59:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/567838418' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 ceph-mon[34061]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43230]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43230]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.602 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43230]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43224]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43224]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43227]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43227]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43227]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.603 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43227]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/567838418' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:54.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:54.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 ceph-mon[33453]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T01:59:54.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41666]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41666]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41666]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41666]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41669]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41669]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41669]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.642 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41669]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/567838418' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 ceph-mon[31245]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53045]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53045]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53045]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53045]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53049]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:54.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53049]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53049]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53049]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53039]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53039]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53039]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53039]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53042]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53042]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53042]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.660 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 01:59:54 smithi016 sudo[53042]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43230]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43233]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43233]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43233]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:54 smithi161 sudo[43233]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41672]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:59:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41672]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41672]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41672]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41675]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:59:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41675]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:59:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41675]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:59:55.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:54 smithi123 sudo[41675]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:59:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T01:59:55.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T01:59:55.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T01:59:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:59:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T01:59:56.425+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2008018770 con 0x7f2018104a30 2024-09-21T01:59:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:57 smithi123 ceph-mon[33453]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:57 smithi123 ceph-mon[33453]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T01:59:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:57 smithi016 ceph-mon[31245]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:57 smithi016 ceph-mon[31245]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T01:59:57.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:57 smithi161 ceph-mon[34061]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:57 smithi161 ceph-mon[34061]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-09-21T01:59:59.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 01:59:59 smithi016 ceph-mon[31245]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 01:59:59 smithi123 ceph-mon[33453]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:59:59.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 01:59:59 smithi161 ceph-mon[34061]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:00 smithi016 ceph-mon[31245]: overall HEALTH_OK 2024-09-21T02:00:00.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:00 smithi123 ceph-mon[33453]: overall HEALTH_OK 2024-09-21T02:00:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:00 smithi161 ceph-mon[34061]: overall HEALTH_OK 2024-09-21T02:00:01.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:01 smithi016 ceph-mon[31245]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:01 smithi123 ceph-mon[33453]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:01.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:01 smithi161 ceph-mon[34061]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:02.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:00:02.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:02.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:02 smithi123 ceph-mon[33453]: Deploying daemon osd.3 on smithi016 2024-09-21T02:00:02.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:00:02.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:02.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:02 smithi016 ceph-mon[31245]: Deploying daemon osd.3 on smithi016 2024-09-21T02:00:02.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T02:00:02.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:02.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:02 smithi161 ceph-mon[34061]: Deploying daemon osd.3 on smithi016 2024-09-21T02:00:03.696 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:03 smithi016 ceph-mon[31245]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:03 smithi123 ceph-mon[33453]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:03.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:03 smithi161 ceph-mon[34061]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:05 smithi123 ceph-mon[33453]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:05 smithi016 ceph-mon[31245]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:05.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:05 smithi161 ceph-mon[34061]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:07.575 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:07 smithi016 ceph-mon[31245]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:07.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:07 smithi123 ceph-mon[33453]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:07.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:07 smithi161 ceph-mon[34061]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:08.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:08.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:08.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:08.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:08.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:08.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:09.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:09 smithi016 ceph-mon[31245]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:09.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:09 smithi123 ceph-mon[33453]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:09.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:09 smithi161 ceph-mon[34061]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:11.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:11.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.349+0000 7f2014ff9640 1 -- 172.21.15.16:0/2915357914 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2018063800 con 0x7f20040b6500 2024-09-21T02:00:12.350 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 msgr2=0x7f20040b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 0x7f20040b8a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2018102440 tx=0x7f2000005e50 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 msgr2=0x7f201819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f200800c970 tx=0x7f200800ce30 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 shutdown_connections 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20040b6500 0x7f20040b8a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201810de60 0x7f201819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2018105380 0x7f201819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 --2- 172.21.15.16:0/2915357914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2018104a30 0x7f201819a390 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 >> 172.21.15.16:0/2915357914 conn(0x7f2018079760 msgr2=0x7f2018100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:00:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 shutdown_connections 2024-09-21T02:00:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:00:12.352+0000 7f201d76f640 1 -- 172.21.15.16:0/2915357914 wait complete. 2024-09-21T02:00:12.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:12 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:12.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:12 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:12 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:12 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:12.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:12 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:12 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:12 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:12.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:12 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:12.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:12 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:13.012 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.3.service 2024-09-21T02:00:13.014 INFO:tasks.cephadm:Deploying osd.4 on smithi123 with /dev/vg_nvme/lv_3... 2024-09-21T02:00:13.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T02:00:13.270 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:00:13.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:13 smithi016 ceph-mon[31245]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:13.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:13 smithi123 ceph-mon[33453]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:13.909 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:00:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:00:13.817+0000 7ff9f4c19640 -1 osd.3 0 log_to_monitors true 2024-09-21T02:00:13.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:13 smithi161 ceph-mon[34061]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:14.562 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:14 smithi016 ceph-mon[31245]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:00:14.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:14 smithi123 ceph-mon[33453]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:00:14.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:14 smithi161 ceph-mon[34061]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T02:00:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-mon[31245]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-mon[31245]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:00:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-mon[31245]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T02:00:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-mon[31245]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:00:15.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:15.759 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:15 smithi123 ceph-mon[33453]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:15.759 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:15 smithi123 ceph-mon[33453]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:00:15.759 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:15 smithi123 ceph-mon[33453]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T02:00:15.759 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:15 smithi123 ceph-mon[33453]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:00:15.759 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:15 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:15.762 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T02:00:15.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-09-21T02:00:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:15 smithi161 ceph-mon[34061]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:15 smithi161 ceph-mon[34061]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T02:00:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:15 smithi161 ceph-mon[34061]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T02:00:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:15 smithi161 ceph-mon[34061]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:00:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:15 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.038 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:00:16.051 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:00:15.938+0000 7ff9f13b0640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T02:00:16.052 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:00:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:00:15.945+0000 7ff9e8991640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 -- 172.21.15.123:0/1910242222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c108cf0 msgr2=0x7fe42c10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 --2- 172.21.15.123:0/1910242222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c108cf0 0x7fe42c10f520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe424009ca0 tx=0x7fe42402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 -- 172.21.15.123:0/1910242222 shutdown_connections 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 --2- 172.21.15.123:0/1910242222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c108cf0 0x7fe42c10f520 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 --2- 172.21.15.123:0/1910242222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 0x7fe42c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 --2- 172.21.15.123:0/1910242222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe42c102a60 0x7fe42c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 -- 172.21.15.123:0/1910242222 >> 172.21.15.123:0/1910242222 conn(0x7fe42c079730 msgr2=0x7fe42c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 -- 172.21.15.123:0/1910242222 shutdown_connections 2024-09-21T02:00:16.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.516+0000 7fe433ab6640 1 -- 172.21.15.123:0/1910242222 wait complete. 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 Processor -- start 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 -- start start 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 0x7fe42c0ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe42c108cf0 0x7fe42c1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c112860 con 0x7fe42c1083a0 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1126e0 con 0x7fe42c102a60 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe433ab6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1129e0 con 0x7fe42c108cf0 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe431852640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe431852640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53946/0 (socket says 172.21.15.123:53946) 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.517+0000 7fe431852640 1 -- 172.21.15.123:0/2391557086 learned_addr learned my addr 172.21.15.123:0/2391557086 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431051640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 0x7fe42c0ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 -- 172.21.15.123:0/2391557086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe42c108cf0 msgr2=0x7fe42c1000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe42c108cf0 0x7fe42c1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 -- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 msgr2=0x7fe42c0ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 0x7fe42c0ffbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 -- 172.21.15.123:0/2391557086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c1a8700 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe431852640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c890 tx=0x7fe41c00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c004220 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe41c007e40 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c1a89f0 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c004a00 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.518+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c1a8f30 con 0x7fe42c102a60 2024-09-21T02:00:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.519+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42c102e70 con 0x7fe42c102a60 2024-09-21T02:00:16.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.521+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fe41c0079b0 con 0x7fe42c102a60 2024-09-21T02:00:16.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.521+0000 7fe422ffd640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 0x7fe4180b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:16.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.522+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3637+0+0 (secure 0 0 0) 0x7fe41c0e6320 con 0x7fe42c102a60 2024-09-21T02:00:16.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.522+0000 7fe431051640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 0x7fe4180b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:16.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.522+0000 7fe431051640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 0x7fe4180b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe428005d80 tx=0x7fe428005c90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:00:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.525+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe41c014030 con 0x7fe42c102a60 2024-09-21T02:00:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:16.713+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe42c063800 con 0x7fe4180b6560 2024-09-21T02:00:16.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:00:16.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T02:00:16.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:16.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:00:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T02:00:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500]' entity='osd.3' 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:16.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:16.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:16.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:17.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:00:17.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:00:17.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi016 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] boot 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:17.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi016 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] boot 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi016 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi016 to 8546M 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: osd.3 [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] boot 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:17.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:19.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:18 smithi123 ceph-mon[33453]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T02:00:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:18 smithi016 ceph-mon[31245]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T02:00:19.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:18 smithi161 ceph-mon[34061]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T02:00:20.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:20.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T02:00:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/4102133413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"}]': finished 2024-09-21T02:00:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T02:00:20.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/4102133413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"}]': finished 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T02:00:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/4102133413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"} : dispatch 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56d0113f-f7b9-463f-b027-41d611c683dc"}]': finished 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T02:00:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:21.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/577571400' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/577571400' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/577571400' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:21.947 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:21 smithi016 ceph-mon[31245]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:21 smithi123 ceph-mon[33453]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:21 smithi161 ceph-mon[34061]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:24.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:23 smithi123 ceph-mon[33453]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-21T02:00:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:23 smithi016 ceph-mon[31245]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-21T02:00:24.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:23 smithi161 ceph-mon[34061]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-21T02:00:25.860 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:25 smithi123 ceph-mon[33453]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T02:00:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:25 smithi016 ceph-mon[31245]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T02:00:26.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:25 smithi161 ceph-mon[34061]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T02:00:27.948 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:27 smithi123 ceph-mon[33453]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:00:27.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:00:27.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:27 smithi016 ceph-mon[31245]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:00:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:00:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:28.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:27 smithi161 ceph-mon[34061]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:00:28.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T02:00:28.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:28.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:28 smithi123 ceph-mon[33453]: Deploying daemon osd.4 on smithi123 2024-09-21T02:00:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:28 smithi016 ceph-mon[31245]: Deploying daemon osd.4 on smithi123 2024-09-21T02:00:29.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:28 smithi161 ceph-mon[34061]: Deploying daemon osd.4 on smithi123 2024-09-21T02:00:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:29 smithi123 ceph-mon[33453]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T02:00:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:29 smithi016 ceph-mon[31245]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T02:00:30.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:29 smithi161 ceph-mon[34061]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T02:00:32.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:31 smithi123 ceph-mon[33453]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T02:00:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:31 smithi016 ceph-mon[31245]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T02:00:32.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:31 smithi161 ceph-mon[34061]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-21T02:00:33.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:33 smithi161 ceph-mon[34061]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:00:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:33 smithi123 ceph-mon[33453]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:00:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:33 smithi016 ceph-mon[31245]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:00:34.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:34.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:34.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:34.984 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:34.985 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:34.985 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:35.942 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:35 smithi123 ceph-mon[33453]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:35 smithi016 ceph-mon[31245]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:35 smithi161 ceph-mon[34061]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:37.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.672+0000 7fe422ffd640 1 -- 172.21.15.123:0/2391557086 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe42c063800 con 0x7fe4180b6560 2024-09-21T02:00:37.673 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 4 on host 'smithi123' 2024-09-21T02:00:37.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 msgr2=0x7fe4180b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:37.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 0x7fe4180b8a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe428005d80 tx=0x7fe428005c90 comp rx=0 tx=0).stop 2024-09-21T02:00:37.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 msgr2=0x7fe42c0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:37.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c890 tx=0x7fe41c00cd50 comp rx=0 tx=0).stop 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 shutdown_connections 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe4180b6560 0x7fe4180b8a70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe42c108cf0 0x7fe42c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe42c1083a0 0x7fe42c0ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 --2- 172.21.15.123:0/2391557086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42c102a60 0x7fe42c0ff670 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 >> 172.21.15.123:0/2391557086 conn(0x7fe42c079730 msgr2=0x7fe42c06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 shutdown_connections 2024-09-21T02:00:37.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:00:37.674+0000 7fe433ab6640 1 -- 172.21.15.123:0/2391557086 wait complete. 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.959 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:37.959 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:37.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:37.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:37.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.293 DEBUG:teuthology.orchestra.run.smithi123:osd.4> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.4.service 2024-09-21T02:00:38.295 INFO:tasks.cephadm:Deploying osd.5 on smithi161 with /dev/vg_nvme/lv_3... 2024-09-21T02:00:38.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T02:00:38.551 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:00:38.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:38.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:38.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:38.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:39 smithi123 ceph-mon[33453]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:39.770 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:00:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:00:39.698+0000 7f58a663d640 -1 osd.4 0 log_to_monitors true 2024-09-21T02:00:39.989 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:39 smithi161 ceph-mon[34061]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:40.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:39 smithi016 ceph-mon[31245]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:40.866 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:40 smithi123 ceph-mon[33453]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:40.866 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:40 smithi123 ceph-mon[33453]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:40.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:40 smithi161 ceph-mon[34061]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:40.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:40 smithi161 ceph-mon[34061]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:41.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:40 smithi016 ceph-mon[31245]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:41.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:40 smithi016 ceph-mon[31245]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T02:00:41.228 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T02:00:41.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi161:vg_nvme/lv_3 2024-09-21T02:00:41.501 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:41.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:41.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 -- 172.21.15.161:0/1742953324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0106280 msgr2=0x7ff3a0106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:41.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 --2- 172.21.15.161:0/1742953324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0106280 0x7ff3a0106690 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff388009930 tx=0x7ff38802a420 comp rx=0 tx=0).stop 2024-09-21T02:00:41.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 -- 172.21.15.161:0/1742953324 shutdown_connections 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 --2- 172.21.15.161:0/1742953324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 0x7ff3a0110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 --2- 172.21.15.161:0/1742953324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0106280 0x7ff3a0106690 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 --2- 172.21.15.161:0/1742953324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0102030 0x7ff3a0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 -- 172.21.15.161:0/1742953324 >> 172.21.15.161:0/1742953324 conn(0x7ff3a0077c60 msgr2=0x7ff3a00780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 -- 172.21.15.161:0/1742953324 shutdown_connections 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.955+0000 7ff3a601e640 1 -- 172.21.15.161:0/1742953324 wait complete. 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.956+0000 7ff3a601e640 1 Processor -- start 2024-09-21T02:00:41.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.956+0000 7ff3a601e640 1 -- start start 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.956+0000 7ff3a601e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.956+0000 7ff3a601e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0106280 0x7ff3a019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.956+0000 7ff3a601e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 0x7ff3a019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff3a601e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a0113950 con 0x7ff3a0106280 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff3a601e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a01137d0 con 0x7ff3a0106bd0 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff3a601e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a0113ad0 con 0x7ff3a0102030 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:41.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42390/0 (socket says 172.21.15.161:42390) 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 -- 172.21.15.161:0/801976985 learned_addr learned my addr 172.21.15.161:0/801976985 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff396ffd640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0106280 0x7ff3a019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39ffff640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 0x7ff3a019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 -- 172.21.15.161:0/801976985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 msgr2=0x7ff3a019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 0x7ff3a019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 -- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0106280 msgr2=0x7ff3a019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0106280 0x7ff3a019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 -- 172.21.15.161:0/801976985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a01a03f0 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39f7fe640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff39000e9c0 tx=0x7ff39000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39000cd80 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a01a06e0 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3900044b0 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.957+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390010430 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.958+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a01a6e40 con 0x7ff3a0102030 2024-09-21T02:00:41.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.959+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a0102440 con 0x7ff3a0102030 2024-09-21T02:00:41.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.960+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff390003f00 con 0x7ff3a0102030 2024-09-21T02:00:41.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.961+0000 7ff39d7fa640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 0x7ff38c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:00:41.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.961+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4036+0+0 (secure 0 0 0) 0x7ff3900e6700 con 0x7ff3a0102030 2024-09-21T02:00:41.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.961+0000 7ff396ffd640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 0x7ff38c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:00:41.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.962+0000 7ff396ffd640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 0x7ff38c0b8af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff388009900 tx=0x7ff3880058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:00:41.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:41.965+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff3900ae0d0 con 0x7ff3a0102030 2024-09-21T02:00:42.020 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:00:41.686+0000 7f58a25bf640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T02:00:42.020 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:00:41.692+0000 7f589e3bd640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:42.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:00:42.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:42.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:00:42.151+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff3a0063800 con 0x7ff38c0b65e0 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi123 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi123 to 8544M 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:42.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi123 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi123 to 8544M 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:42.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:42.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi123 2024-09-21T02:00:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi123 to 8544M 2024-09-21T02:00:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:00:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T02:00:43.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:43.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:43.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:00:43.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:00:43.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:00:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:00:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] boot 2024-09-21T02:00:44.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T02:00:44.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.058 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] boot 2024-09-21T02:00:44.059 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T02:00:44.059 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 131 MiB used, 357 GiB / 358 GiB avail 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: osd.4 [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] boot 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T02:00:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T02:00:44.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T02:00:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/2885207947' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"}]': finished 2024-09-21T02:00:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T02:00:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/2885207947' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"}]': finished 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T02:00:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/2885207947' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"} : dispatch 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d1bb4b0-fc3a-488e-8b68-41360a19cef0"}]': finished 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T02:00:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:00:46.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:45 smithi123 ceph-mon[33453]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:46.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:45 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/1409370715' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:45 smithi016 ceph-mon[31245]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:45 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/1409370715' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:45 smithi161 ceph-mon[34061]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:45 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/1409370715' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:00:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:47 smithi123 ceph-mon[33453]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:48.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:47 smithi016 ceph-mon[31245]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:48.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:47 smithi161 ceph-mon[34061]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:49.960 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:49 smithi161 ceph-mon[34061]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:50.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:49 smithi123 ceph-mon[33453]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:49 smithi016 ceph-mon[31245]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:51.753 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:51 smithi161 ceph-mon[34061]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:51 smithi123 ceph-mon[33453]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:52.057 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:51 smithi016 ceph-mon[31245]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:53 smithi123 ceph-mon[33453]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:00:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:53 smithi123 ceph-mon[33453]: Deploying daemon osd.5 on smithi161 2024-09-21T02:00:54.146 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:53 smithi161 ceph-mon[34061]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:54.147 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:00:54.147 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:54.147 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:53 smithi161 ceph-mon[34061]: Deploying daemon osd.5 on smithi161 2024-09-21T02:00:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:53 smithi016 ceph-mon[31245]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T02:00:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:00:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:53 smithi016 ceph-mon[31245]: Deploying daemon osd.5 on smithi161 2024-09-21T02:00:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:55 smithi123 ceph-mon[33453]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:56.149 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:55 smithi161 ceph-mon[34061]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:55 smithi016 ceph-mon[31245]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:57 smithi161 ceph-mon[34061]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:58.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:57 smithi123 ceph-mon[33453]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:57 smithi016 ceph-mon[31245]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:59 smithi161 ceph-mon[34061]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:00:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:00:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:00:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:00:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:59 smithi123 ceph-mon[33453]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:00:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:59 smithi016 ceph-mon[31245]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:00:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:01.755 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:01 smithi161 ceph-mon[34061]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:02.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:01 smithi123 ceph-mon[33453]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:01 smithi016 ceph-mon[31245]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:02 smithi016 ceph-mon[31245]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.159 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:02 smithi161 ceph-mon[34061]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:03.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.200+0000 7ff39d7fa640 1 -- 172.21.15.161:0/801976985 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff3a0063800 con 0x7ff38c0b65e0 2024-09-21T02:01:03.202 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 5 on host 'smithi161' 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 msgr2=0x7ff38c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 0x7ff38c0b8af0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff388009900 tx=0x7ff3880058d0 comp rx=0 tx=0).stop 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 msgr2=0x7ff3a019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff39000e9c0 tx=0x7ff39000ee80 comp rx=0 tx=0).stop 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 shutdown_connections 2024-09-21T02:01:03.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff38c0b65e0 0x7ff38c0b8af0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a0106bd0 0x7ff3a019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a0106280 0x7ff3a019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 --2- 172.21.15.161:0/801976985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3a0102030 0x7ff3a019a480 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 >> 172.21.15.161:0/801976985 conn(0x7ff3a0077c60 msgr2=0x7ff3a0107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 shutdown_connections 2024-09-21T02:01:03.205 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:03.203+0000 7ff3a601e640 1 -- 172.21.15.161:0/801976985 wait complete. 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:02 smithi123 ceph-mon[33453]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:03.897 DEBUG:teuthology.orchestra.run.smithi161:osd.5> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.5.service 2024-09-21T02:01:03.899 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-09-21T02:01:03.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:01:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:01:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:05.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:04 smithi161 ceph-mon[34061]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:05.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:04 smithi123 ceph-mon[33453]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:05.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:04 smithi016 ceph-mon[31245]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:05.912 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:05 smithi161 ceph-mon[34061]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:05.912 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:05 smithi161 ceph-mon[34061]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:05.912 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:01:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:01:05.558+0000 7fe75f45a640 -1 osd.5 0 log_to_monitors true 2024-09-21T02:01:06.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:05 smithi016 ceph-mon[31245]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:06.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:05 smithi016 ceph-mon[31245]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:05 smithi123 ceph-mon[33453]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:05 smithi123 ceph-mon[33453]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T02:01:06.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:01:06.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:07.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:07.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:01:07.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T02:01:07.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:07.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:07.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:01:07.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T02:01:07.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T02:01:07.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:07.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:07.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:07.664 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:01:07.364+0000 7fe75b3dc640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T02:01:07.665 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:01:07.374+0000 7fe7531d2640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 -- 172.21.15.16:0/3577624057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc924108d20 msgr2=0x7fc92410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 --2- 172.21.15.16:0/3577624057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc924108d20 0x7fc92410f550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009900 tx=0x7fc91c02a410 comp rx=0 tx=0).stop 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 -- 172.21.15.16:0/3577624057 shutdown_connections 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 --2- 172.21.15.16:0/3577624057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc924108d20 0x7fc92410f550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 --2- 172.21.15.16:0/3577624057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9241083d0 0x7fc9241087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 --2- 172.21.15.16:0/3577624057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924102ae0 0x7fc924102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 -- 172.21.15.16:0/3577624057 >> 172.21.15.16:0/3577624057 conn(0x7fc924079760 msgr2=0x7fc924079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:07.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 -- 172.21.15.16:0/3577624057 shutdown_connections 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.911+0000 7fc92be42640 1 -- 172.21.15.16:0/3577624057 wait complete. 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 Processor -- start 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 -- start start 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 0x7fc92419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 0x7fc9241a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc924110b10 con 0x7fc9241083d0 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc924110990 con 0x7fc924108d20 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92be42640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc924110c90 con 0x7fc924102ae0 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41912/0 (socket says 172.21.15.16:41912) 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 -- 172.21.15.16:0/2180492528 learned_addr learned my addr 172.21.15.16:0/2180492528 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc92a3df640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 0x7fc9241a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:07.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc929bde640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 0x7fc92419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 -- 172.21.15.16:0/2180492528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 msgr2=0x7fc92419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 0x7fc92419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 -- 172.21.15.16:0/2180492528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 msgr2=0x7fc9241a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 0x7fc9241a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.912+0000 7fc9293dd640 1 -- 172.21.15.16:0/2180492528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9241a2490 con 0x7fc9241083d0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc92a3df640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 0x7fc9241a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc929bde640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 0x7fc92419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc9293dd640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc91400d740 tx=0x7fc91400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914017070 con 0x7fc9241083d0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc91400de50 con 0x7fc9241083d0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9241a2780 con 0x7fc9241083d0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914004c60 con 0x7fc9241083d0 2024-09-21T02:01:07.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.913+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9241a8ee0 con 0x7fc9241083d0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.914+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc924103d10 con 0x7fc9241083d0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.916+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fc914004e00 con 0x7fc9241083d0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.916+0000 7fc91affd640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 0x7fc9100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.917+0000 7fc929bde640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 0x7fc9100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.917+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4468+0+0 (secure 0 0 0) 0x7fc9140e6230 con 0x7fc9241083d0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.917+0000 7fc929bde640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 0x7fc9100b8a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc920005fd0 tx=0x7fc920009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:07.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:07.920+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc9140ad9c0 con 0x7fc9241083d0 2024-09-21T02:01:08.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:08.121+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc924101c70 con 0x7fc9100b6560 2024-09-21T02:01:08.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi161 2024-09-21T02:01:08.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi161 to 8544M 2024-09-21T02:01:08.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:01:08.122 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T02:01:08.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.123 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:07 smithi016 ceph-mon[31245]: from='osd.5 ' entity='osd.5' 2024-09-21T02:01:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi161 2024-09-21T02:01:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi161 to 8544M 2024-09-21T02:01:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:01:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T02:01:08.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:07 smithi123 ceph-mon[33453]: from='osd.5 ' entity='osd.5' 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi161 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi161 to 8544M 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:07 smithi161 ceph-mon[34061]: from='osd.5 ' entity='osd.5' 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] boot 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:08 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:01:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:01:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] boot 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:08 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: osd.5 [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] boot 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:09.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:09.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:08 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:10.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:09 smithi016 ceph-mon[31245]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T02:01:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:09 smithi123 ceph-mon[33453]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T02:01:10.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:09 smithi161 ceph-mon[34061]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T02:01:11.279 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T02:01:11.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:11.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"} : dispatch 2024-09-21T02:01:11.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"}]': finished 2024-09-21T02:01:11.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T02:01:11.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:10 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:11.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T02:01:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"} : dispatch 2024-09-21T02:01:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"}]': finished 2024-09-21T02:01:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T02:01:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:10 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"} : dispatch 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3176116285' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c00ece7-8034-440b-ba71-50cc66f698a1"}]': finished 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T02:01:11.417 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:10 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/764663259' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/764663259' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:12.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/764663259' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:13.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:12 smithi123 ceph-mon[33453]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-09-21T02:01:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:12 smithi016 ceph-mon[31245]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-09-21T02:01:13.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:12 smithi161 ceph-mon[34061]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-09-21T02:01:15.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:15 smithi123 ceph-mon[33453]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:01:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:15 smithi016 ceph-mon[31245]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:01:15.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:15 smithi161 ceph-mon[34061]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:01:17.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:17 smithi016 ceph-mon[31245]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:01:17.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:17 smithi161 ceph-mon[34061]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:01:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:17 smithi123 ceph-mon[33453]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T02:01:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:19 smithi016 ceph-mon[31245]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:01:19.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:19 smithi161 ceph-mon[34061]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:01:19.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:19 smithi123 ceph-mon[33453]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T02:01:20.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:01:20.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:20.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:20 smithi016 ceph-mon[31245]: Deploying daemon osd.6 on smithi016 2024-09-21T02:01:20.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:01:20.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:20.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:20 smithi161 ceph-mon[34061]: Deploying daemon osd.6 on smithi016 2024-09-21T02:01:20.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T02:01:20.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:20.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:20 smithi123 ceph-mon[33453]: Deploying daemon osd.6 on smithi016 2024-09-21T02:01:21.189 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:21 smithi016 ceph-mon[31245]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:01:21.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:21 smithi161 ceph-mon[34061]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:01:21.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:21 smithi123 ceph-mon[33453]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:01:23.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:23 smithi016 ceph-mon[31245]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:01:23.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:23 smithi161 ceph-mon[34061]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:01:23.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:23 smithi123 ceph-mon[33453]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T02:01:25.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:25 smithi016 ceph-mon[31245]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:25.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:25 smithi161 ceph-mon[34061]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:25.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:25 smithi123 ceph-mon[33453]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:26.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:26.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:26.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:27.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:27 smithi016 ceph-mon[31245]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:27 smithi161 ceph-mon[34061]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:27 smithi123 ceph-mon[33453]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:29.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:29.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:29.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:29.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:29.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:29.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.987+0000 7fc91affd640 1 -- 172.21.15.16:0/2180492528 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc924101c70 con 0x7fc9100b6560 2024-09-21T02:01:29.988 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 msgr2=0x7fc9100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 0x7fc9100b8a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc920005fd0 tx=0x7fc920009500 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 msgr2=0x7fc92419cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc91400d740 tx=0x7fc91400dc00 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 shutdown_connections 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc9100b6560 0x7fc9100b8a70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc924108d20 0x7fc9241a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9241083d0 0x7fc92419cac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 --2- 172.21.15.16:0/2180492528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc924102ae0 0x7fc92419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 >> 172.21.15.16:0/2180492528 conn(0x7fc924079760 msgr2=0x7fc924101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 shutdown_connections 2024-09-21T02:01:29.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:01:29.990+0000 7fc92be42640 1 -- 172.21.15.16:0/2180492528 wait complete. 2024-09-21T02:01:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:30.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:30.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:30.843 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.6.service 2024-09-21T02:01:30.846 INFO:tasks.cephadm:Deploying osd.7 on smithi123 with /dev/vg_nvme/lv_2... 2024-09-21T02:01:30.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:01:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:31.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:31.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:31.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:31.100 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:01:31.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:31.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:31.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:31 smithi123 ceph-mon[33453]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:32.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:31 smithi016 ceph-mon[31245]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:32.090 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:01:31 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:01:31.945+0000 7f2a85fd8640 -1 osd.6 0 log_to_monitors true 2024-09-21T02:01:32.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:31 smithi161 ceph-mon[34061]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:32.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:32 smithi123 ceph-mon[33453]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:01:33.066 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:32 smithi161 ceph-mon[34061]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:01:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:32 smithi016 ceph-mon[31245]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T02:01:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T02:01:33.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-09-21T02:01:33.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:33 smithi016 ceph-mon[31245]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:33.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:33 smithi016 ceph-mon[31245]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:01:33.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:33 smithi016 ceph-mon[31245]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T02:01:33.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:33 smithi016 ceph-mon[31245]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:01:33.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:33 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:01:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:33 smithi123 ceph-mon[33453]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:33 smithi123 ceph-mon[33453]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:01:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:33 smithi123 ceph-mon[33453]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T02:01:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:33 smithi123 ceph-mon[33453]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:01:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:33 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:34.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:33 smithi161 ceph-mon[34061]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:33 smithi161 ceph-mon[34061]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T02:01:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:33 smithi161 ceph-mon[34061]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T02:01:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:33 smithi161 ceph-mon[34061]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:01:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:33 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:34.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 -- 172.21.15.123:0/566563931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954102a60 msgr2=0x7f5954102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 --2- 172.21.15.123:0/566563931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954102a60 0x7f5954102e70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5944009900 tx=0x7f594402a410 comp rx=0 tx=0).stop 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 -- 172.21.15.123:0/566563931 shutdown_connections 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 --2- 172.21.15.123:0/566563931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954108cf0 0x7f595410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 --2- 172.21.15.123:0/566563931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 --2- 172.21.15.123:0/566563931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954102a60 0x7f5954102e70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 -- 172.21.15.123:0/566563931 >> 172.21.15.123:0/566563931 conn(0x7f5954079730 msgr2=0x7f5954079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 -- 172.21.15.123:0/566563931 shutdown_connections 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.357+0000 7f595c522640 1 -- 172.21.15.123:0/566563931 wait complete. 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 Processor -- start 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 -- start start 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954102a60 0x7f595410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:34.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5954113930 con 0x7f5954102a60 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59541137b0 con 0x7f5954108cf0 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595c522640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5954113ab0 con 0x7f59541083a0 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595aabf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f5959abd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595aabf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39308/0 (socket says 172.21.15.123:39308) 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595a2be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954102a60 0x7f595410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.358+0000 7f595aabf640 1 -- 172.21.15.123:0/2300569934 learned_addr learned my addr 172.21.15.123:0/2300569934 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 -- 172.21.15.123:0/2300569934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 msgr2=0x7f5954110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:34.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 -- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954102a60 msgr2=0x7f595410ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954102a60 0x7f595410ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 -- 172.21.15.123:0/2300569934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59541a8660 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f595aabf640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f5959abd640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541104f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f594800d6a0 tx=0x7f594800db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948017070 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59541a8950 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5948009820 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948004bc0 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.359+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59541a8e90 con 0x7f59541083a0 2024-09-21T02:01:34.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.360+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5954102e70 con 0x7f59541083a0 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.362+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f5948005220 con 0x7f59541083a0 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.362+0000 7f59537fe640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 0x7f593c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.363+0000 7f595a2be640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 0x7f593c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.363+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4900+0+0 (secure 0 0 0) 0x7f59480e5ed0 con 0x7f59541083a0 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.363+0000 7f595a2be640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 0x7f593c0b8a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5954111a00 tx=0x7f59440058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:34.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.366+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f59480b1870 con 0x7f59541083a0 2024-09-21T02:01:34.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:34.554+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5954063800 con 0x7f593c0b6560 2024-09-21T02:01:34.658 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:01:34.220+0000 7f2a81f5a640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T02:01:34.658 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:01:34.229+0000 7f2a79d50640 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:01:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:01:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T02:01:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' 2024-09-21T02:01:35.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:35.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:35.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:01:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T02:01:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:35.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955]' entity='osd.6' 2024-09-21T02:01:35.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:01:35.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:01:35.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:35.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:01:35.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:01:35.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi016 2024-09-21T02:01:35.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-21T02:01:35.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:35.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: from='client.24460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:35.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:35.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] boot 2024-09-21T02:01:35.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T02:01:35.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:01:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:01:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi016 2024-09-21T02:01:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-21T02:01:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:36.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: from='client.24460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:36.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:36.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] boot 2024-09-21T02:01:36.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T02:01:36.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:01:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:01:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi016 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi016 to 5697M 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: from='client.24460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: osd.6 [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] boot 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T02:01:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T02:01:38.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/3780017485' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"}]': finished 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:01:38.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:37 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/2643596724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/3780017485' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"}]': finished 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:01:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:37 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/2643596724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:38.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/3780017485' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"} : dispatch 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15352f53-b24f-4c15-8405-1971628489a7"}]': finished 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:01:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:37 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/2643596724' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:01:40.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:39 smithi123 ceph-mon[33453]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:39 smithi016 ceph-mon[31245]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:40.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:39 smithi161 ceph-mon[34061]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:41 smithi123 ceph-mon[33453]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:41 smithi016 ceph-mon[31245]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:42.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:41 smithi161 ceph-mon[34061]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:43 smithi123 ceph-mon[33453]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:43 smithi016 ceph-mon[31245]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:44.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:43 smithi161 ceph-mon[34061]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:46.018 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:45 smithi123 ceph-mon[33453]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:46.018 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:45 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:01:46.019 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:45 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:45 smithi016 ceph-mon[31245]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:45 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:01:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:45 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:45 smithi161 ceph-mon[34061]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:46.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:45 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T02:01:46.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:45 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:01:46.974 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:46 smithi123 ceph-mon[33453]: Deploying daemon osd.7 on smithi123 2024-09-21T02:01:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:46 smithi016 ceph-mon[31245]: Deploying daemon osd.7 on smithi123 2024-09-21T02:01:47.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:46 smithi161 ceph-mon[34061]: Deploying daemon osd.7 on smithi123 2024-09-21T02:01:47.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:47 smithi123 ceph-mon[33453]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:48.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:47 smithi016 ceph-mon[31245]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:48.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:47 smithi161 ceph-mon[34061]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:49 smithi016 ceph-mon[31245]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:50.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:49 smithi161 ceph-mon[34061]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:50.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:49 smithi123 ceph-mon[33453]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:51 smithi123 ceph-mon[33453]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:51 smithi016 ceph-mon[31245]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:52.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:51 smithi161 ceph-mon[34061]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:52.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:01:52.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:52.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:53 smithi123 ceph-mon[33453]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:54.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:53 smithi016 ceph-mon[31245]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:54.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:53 smithi161 ceph-mon[34061]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:55.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.188+0000 7f59537fe640 1 -- 172.21.15.123:0/2300569934 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5954063800 con 0x7f593c0b6560 2024-09-21T02:01:55.190 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 7 on host 'smithi123' 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 msgr2=0x7f593c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 0x7f593c0b8a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5954111a00 tx=0x7f59440058d0 comp rx=0 tx=0).stop 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 msgr2=0x7f59541104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541104f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f594800d6a0 tx=0x7f594800db60 comp rx=0 tx=0).stop 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 shutdown_connections 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f593c0b6560 0x7f593c0b8a70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5954108cf0 0x7f5954110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59541083a0 0x7f59541104f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:55.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 --2- 172.21.15.123:0/2300569934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5954102a60 0x7f595410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:55.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.191+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 >> 172.21.15.123:0/2300569934 conn(0x7f5954079730 msgr2=0x7f5954101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:55.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.192+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 shutdown_connections 2024-09-21T02:01:55.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:01:55.192+0000 7f595c522640 1 -- 172.21.15.123:0/2300569934 wait complete. 2024-09-21T02:01:55.848 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:55 smithi123 ceph-mon[33453]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:55.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:55.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:55.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:55.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:55.876 DEBUG:teuthology.orchestra.run.smithi123:osd.7> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.7.service 2024-09-21T02:01:55.878 INFO:tasks.cephadm:Deploying osd.8 on smithi161 with /dev/vg_nvme/lv_2... 2024-09-21T02:01:55.878 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T02:01:56.133 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:01:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:55 smithi016 ceph-mon[31245]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:55 smithi161 ceph-mon[34061]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:56.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:01:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:57 smithi161 ceph-mon[34061]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:57 smithi016 ceph-mon[31245]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:58.181 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:57 smithi123 ceph-mon[33453]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:01:58.181 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:01:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:01:57.845+0000 7f749cce5640 -1 osd.7 0 log_to_monitors true 2024-09-21T02:01:58.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T02:01:58.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi161:vg_nvme/lv_2 2024-09-21T02:01:59.072 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:01:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:58 smithi016 ceph-mon[31245]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:58 smithi016 ceph-mon[31245]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:58 smithi161 ceph-mon[34061]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:58 smithi161 ceph-mon[34061]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:58 smithi123 ceph-mon[33453]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:58 smithi123 ceph-mon[33453]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T02:01:59.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 -- 172.21.15.161:0/2407800748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea01094b0 msgr2=0x7f7ea01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:59.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/2407800748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea01094b0 0x7f7ea01098c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009900 tx=0x7f7e8802a410 comp rx=0 tx=0).stop 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 -- 172.21.15.161:0/2407800748 shutdown_connections 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/2407800748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0109e00 0x7f7ea0110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/2407800748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea01094b0 0x7f7ea01098c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/2407800748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea0103bc0 0x7f7ea0103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 -- 172.21.15.161:0/2407800748 >> 172.21.15.161:0/2407800748 conn(0x7f7ea007b910 msgr2=0x7f7ea007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 -- 172.21.15.161:0/2407800748 shutdown_connections 2024-09-21T02:01:59.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.585+0000 7f7ea4eac640 1 -- 172.21.15.161:0/2407800748 wait complete. 2024-09-21T02:01:59.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.586+0000 7f7ea4eac640 1 Processor -- start 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.586+0000 7f7ea4eac640 1 -- start start 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0103bc0 0x7f7ea019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea01094b0 0x7f7ea019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea0113970 con 0x7f7ea01094b0 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea01137f0 con 0x7f7ea0103bc0 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7ea4eac640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea0113af0 con 0x7f7ea0109e00 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40054/0 (socket says 172.21.15.161:40054) 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 -- 172.21.15.161:0/3569812466 learned_addr learned my addr 172.21.15.161:0/3569812466 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9e59c640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0103bc0 0x7f7ea019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e97fff640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea01094b0 0x7f7ea019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 -- 172.21.15.161:0/3569812466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0103bc0 msgr2=0x7f7ea019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:59.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0103bc0 0x7f7ea019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 -- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea01094b0 msgr2=0x7f7ea019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea01094b0 0x7f7ea019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 -- 172.21.15.161:0/3569812466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea01a03b0 con 0x7f7ea0109e00 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.587+0000 7f7e9ed9d640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800eb10 tx=0x7f7e9800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.588+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98004b20 con 0x7f7ea0109e00 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.588+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea01a06a0 con 0x7f7ea0109e00 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.588+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e98004cc0 con 0x7f7ea0109e00 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.588+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98010690 con 0x7f7ea0109e00 2024-09-21T02:01:59.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.588+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea01a6e00 con 0x7f7ea0109e00 2024-09-21T02:01:59.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.589+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea0103fd0 con 0x7f7ea0109e00 2024-09-21T02:01:59.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.590+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7e98005000 con 0x7f7ea0109e00 2024-09-21T02:01:59.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.591+0000 7f7e977fe640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 0x7f7e8c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:01:59.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.591+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5316+0+0 (secure 0 0 0) 0x7f7e980e7030 con 0x7f7ea0109e00 2024-09-21T02:01:59.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.591+0000 7f7e9e59c640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 0x7f7e8c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:01:59.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.592+0000 7f7e9e59c640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 0x7f7e8c0b8a70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009b60 tx=0x7f7e90005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:01:59.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.594+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7e98056710 con 0x7f7ea0109e00 2024-09-21T02:01:59.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:01:59.780+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7ea0063800 con 0x7f7e8c0b6560 2024-09-21T02:02:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:01:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.161 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:00.162 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:00.163 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:01:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.248 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:00.248 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.249 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:00.250 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:01:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:00.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:02:00.247+0000 7f749947c640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T02:02:00.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:02:00.256+0000 7f7490a5d640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:02:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi123 2024-09-21T02:02:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi123 to 5696M 2024-09-21T02:02:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:02:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='osd.7 ' entity='osd.7' 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] boot 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:02:01.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi123 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi123 to 5696M 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:01.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='osd.7 ' entity='osd.7' 2024-09-21T02:02:01.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] boot 2024-09-21T02:02:01.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:02:01.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi123 2024-09-21T02:02:01.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi123 to 5696M 2024-09-21T02:02:01.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:01.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 626 GiB / 626 GiB avail 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='osd.7 ' entity='osd.7' 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: osd.7 [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] boot 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T02:02:01.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T02:02:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:01 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:02:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:01 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:02:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:01 smithi016 ceph-mon[31245]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T02:02:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:01 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:02:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:01 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:02:02.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:01 smithi161 ceph-mon[34061]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T02:02:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:01 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:02:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:01 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:02:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:01 smithi123 ceph-mon[33453]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/3981537582' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"}]': finished 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:02:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:02 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:03.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:03.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/3981537582' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"}]': finished 2024-09-21T02:02:03.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:02:03.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:02 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/3981537582' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"} : dispatch 2024-09-21T02:02:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71da0d14-803c-47fa-9a77-e44887e81da4"}]': finished 2024-09-21T02:02:03.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: osdmap e51: 9 total, 8 up, 9 in 2024-09-21T02:02:03.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:02 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/961804061' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/961804061' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/961804061' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:04 smithi016 ceph-mon[31245]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:05.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:04 smithi161 ceph-mon[34061]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:05.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:04 smithi123 ceph-mon[33453]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:07.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:07 smithi123 ceph-mon[33453]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:07.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:07 smithi016 ceph-mon[31245]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:07.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:07 smithi161 ceph-mon[34061]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:09 smithi161 ceph-mon[34061]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:09 smithi016 ceph-mon[31245]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:09 smithi123 ceph-mon[33453]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:11.377 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:11 smithi161 ceph-mon[34061]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T02:02:11.377 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:02:11.377 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:11 smithi016 ceph-mon[31245]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T02:02:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:02:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:11 smithi123 ceph-mon[33453]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T02:02:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-21T02:02:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:12.327 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:12 smithi161 ceph-mon[34061]: Deploying daemon osd.8 on smithi161 2024-09-21T02:02:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:12 smithi016 ceph-mon[31245]: Deploying daemon osd.8 on smithi161 2024-09-21T02:02:12.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:12 smithi123 ceph-mon[33453]: Deploying daemon osd.8 on smithi161 2024-09-21T02:02:13.322 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:13 smithi161 ceph-mon[34061]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:13 smithi016 ceph-mon[31245]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:13 smithi123 ceph-mon[33453]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:15.305 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:15 smithi161 ceph-mon[34061]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:15.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:15 smithi016 ceph-mon[31245]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:15.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:15 smithi123 ceph-mon[33453]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:17.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:17 smithi161 ceph-mon[34061]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:17.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:17.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:17.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:17 smithi016 ceph-mon[31245]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:17 smithi123 ceph-mon[33453]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:19 smithi161 ceph-mon[34061]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:19 smithi016 ceph-mon[31245]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:19.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:19 smithi123 ceph-mon[33453]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:21.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.015+0000 7f7e977fe640 1 -- 172.21.15.161:0/3569812466 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7ea0063800 con 0x7f7e8c0b6560 2024-09-21T02:02:21.016 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 8 on host 'smithi161' 2024-09-21T02:02:21.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 msgr2=0x7f7e8c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:21.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 0x7f7e8c0b8a70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009b60 tx=0x7f7e90005f90 comp rx=0 tx=0).stop 2024-09-21T02:02:21.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 msgr2=0x7f7ea019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800eb10 tx=0x7f7e9800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 shutdown_connections 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6560 0x7f7e8c0b8a70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea0109e00 0x7f7ea019fc80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea01094b0 0x7f7ea019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 --2- 172.21.15.161:0/3569812466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea0103bc0 0x7f7ea019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 >> 172.21.15.161:0/3569812466 conn(0x7f7ea007b910 msgr2=0x7f7ea01076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 shutdown_connections 2024-09-21T02:02:21.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:02:21.018+0000 7f7ea4eac640 1 -- 172.21.15.161:0/3569812466 wait complete. 2024-09-21T02:02:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:20 smithi016 ceph-mon[31245]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:20 smithi161 ceph-mon[34061]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:20 smithi123 ceph-mon[33453]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:21.771 DEBUG:teuthology.orchestra.run.smithi161:osd.8> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.8.service 2024-09-21T02:02:21.773 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-09-21T02:02:21.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:02:22.003 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:22.003 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:22.003 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:21 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:22.046 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:02:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:21 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:21 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:23.102 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:22 smithi161 ceph-mon[34061]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:22 smithi016 ceph-mon[31245]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:22 smithi123 ceph-mon[33453]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:23.395 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:02:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:02:23.245+0000 7ff402b9b640 -1 osd.8 0 log_to_monitors true 2024-09-21T02:02:24.156 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:23 smithi161 ceph-mon[34061]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.156 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:23 smithi161 ceph-mon[34061]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:23 smithi016 ceph-mon[31245]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:23 smithi016 ceph-mon[31245]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:23 smithi123 ceph-mon[33453]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:23 smithi123 ceph-mon[33453]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-21T02:02:24.770 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:02:24.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-09-21T02:02:25.064 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:24 smithi016 ceph-mon[31245]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:02:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:02:25.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:25.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:24 smithi123 ceph-mon[33453]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:25.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:02:24.924+0000 7ff3feb1d640 -1 osd.8 0 waiting for initial osdmap 2024-09-21T02:02:25.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:02:24.938+0000 7ff3fa91b640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: from='osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: osdmap e52: 9 total, 8 up, 9 in 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:02:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:24 smithi161 ceph-mon[34061]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 715 GiB / 715 GiB avail 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 -- 172.21.15.16:0/31345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c1094f0 msgr2=0x7f3c1c109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 --2- 172.21.15.16:0/31345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c109900 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009900 tx=0x7f3c1002a410 comp rx=0 tx=0).stop 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 -- 172.21.15.16:0/31345459 shutdown_connections 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 --2- 172.21.15.16:0/31345459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c109e40 0x7f3c1c110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 --2- 172.21.15.16:0/31345459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c109900 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 --2- 172.21.15.16:0/31345459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:25.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.608+0000 7f3c21492640 1 -- 172.21.15.16:0/31345459 >> 172.21.15.16:0/31345459 conn(0x7f3c1c0ff600 msgr2=0x7f3c1c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:02:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.609+0000 7f3c21492640 1 -- 172.21.15.16:0/31345459 shutdown_connections 2024-09-21T02:02:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.609+0000 7f3c21492640 1 -- 172.21.15.16:0/31345459 wait complete. 2024-09-21T02:02:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.609+0000 7f3c21492640 1 Processor -- start 2024-09-21T02:02:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.609+0000 7f3c21492640 1 -- start start 2024-09-21T02:02:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c1139b0 con 0x7f3c1c109e40 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c113830 con 0x7f3c1c1094f0 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c21492640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c113b30 con 0x7f3c1c103bb0 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56738/0 (socket says 172.21.15.16:56738) 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 -- 172.21.15.16:0/1140983708 learned_addr learned my addr 172.21.15.16:0/1140983708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1affd640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1a7fc640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 -- 172.21.15.16:0/1140983708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 msgr2=0x7f3c1c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 -- 172.21.15.16:0/1140983708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c1094f0 msgr2=0x7f3c1c19aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c19aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 -- 172.21.15.16:0/1140983708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c1a0410 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1affd640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.610+0000 7f3c1b7fe640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400eb10 tx=0x7f3c1400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.611+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14004b20 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.611+0000 7f3c21492640 1 -- 172.21.15.16:0/1140983708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1c1a0700 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.611+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c14004cc0 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.611+0000 7f3c21492640 1 -- 172.21.15.16:0/1140983708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1c1a6e60 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.611+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14010690 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.612+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc005290 con 0x7f3c1c109e40 2024-09-21T02:02:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.614+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f3c14005000 con 0x7f3c1c109e40 2024-09-21T02:02:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.615+0000 7f3c03fff640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 0x7f3c040b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.615+0000 7f3c1affd640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 0x7f3c040b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.615+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5764+0+0 (secure 0 0 0) 0x7f3c140e7130 con 0x7f3c1c109e40 2024-09-21T02:02:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.615+0000 7f3c1affd640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 0x7f3c040b8a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0058c0 tx=0x7f3c0c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:02:25.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.618+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c14056710 con 0x7f3c1c109e40 2024-09-21T02:02:25.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:25.818+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3bfc000bc0 con 0x7f3c040b6560 2024-09-21T02:02:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:02:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T02:02:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi161 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi161 to 5696M 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] boot 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:02:26.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:26.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:26.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:25 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi161 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi161 to 5696M 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] boot 2024-09-21T02:02:26.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:02:26.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:26.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:26.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:25 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: osdmap e53: 9 total, 8 up, 9 in 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi161 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi161 to 5696M 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: osd.8 [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] boot 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: osdmap e54: 9 total, 9 up, 9 in 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:26.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:25 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:27.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:26 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:02:27.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:26 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:02:27.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:26 smithi016 ceph-mon[31245]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:27.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:26 smithi016 ceph-mon[31245]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:27.204 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:26 smithi016 ceph-mon[31245]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:02:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:26 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:02:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:26 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:02:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:26 smithi123 ceph-mon[33453]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:26 smithi123 ceph-mon[33453]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:26 smithi123 ceph-mon[33453]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:02:27.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:26 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:02:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:26 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:02:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:26 smithi161 ceph-mon[34061]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:26 smithi161 ceph-mon[34061]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:26 smithi161 ceph-mon[34061]: osdmap e55: 9 total, 9 up, 9 in 2024-09-21T02:02:28.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:28 smithi016 ceph-mon[31245]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T02:02:28.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"} : dispatch 2024-09-21T02:02:28.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"}]': finished 2024-09-21T02:02:28.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:28 smithi016 ceph-mon[31245]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:02:28.899 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:28 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:28 smithi123 ceph-mon[33453]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T02:02:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"} : dispatch 2024-09-21T02:02:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"}]': finished 2024-09-21T02:02:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:28 smithi123 ceph-mon[33453]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:02:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:28 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:29.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:28 smithi161 ceph-mon[34061]: osdmap e56: 9 total, 9 up, 9 in 2024-09-21T02:02:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"} : dispatch 2024-09-21T02:02:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1338853349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a8fc26-a3a8-4a74-97d2-68218278bb36"}]': finished 2024-09-21T02:02:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:28 smithi161 ceph-mon[34061]: osdmap e57: 10 total, 9 up, 10 in 2024-09-21T02:02:29.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:28 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:29 smithi123 ceph-mon[33453]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:29 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2826647716' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:29 smithi016 ceph-mon[31245]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:29 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2826647716' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:29 smithi161 ceph-mon[34061]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:30.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:29 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2826647716' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:31 smithi123 ceph-mon[33453]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:31 smithi016 ceph-mon[31245]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:32.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:31 smithi161 ceph-mon[34061]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:33.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:33 smithi016 ceph-mon[31245]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:33.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:33 smithi161 ceph-mon[34061]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:33 smithi123 ceph-mon[33453]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:02:35.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:35 smithi016 ceph-mon[31245]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:35 smithi123 ceph-mon[33453]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:36.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:35 smithi161 ceph-mon[34061]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T02:02:37.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:37 smithi016 ceph-mon[31245]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T02:02:37.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:02:37.897 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:38.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:37 smithi123 ceph-mon[33453]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T02:02:38.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:02:38.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:37 smithi161 ceph-mon[34061]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T02:02:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-21T02:02:38.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:38 smithi016 ceph-mon[31245]: Deploying daemon osd.9 on smithi016 2024-09-21T02:02:39.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:38 smithi123 ceph-mon[33453]: Deploying daemon osd.9 on smithi016 2024-09-21T02:02:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:38 smithi161 ceph-mon[34061]: Deploying daemon osd.9 on smithi016 2024-09-21T02:02:39.827 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:39 smithi016 ceph-mon[31245]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:40.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:39 smithi123 ceph-mon[33453]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:40.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:39 smithi161 ceph-mon[34061]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T02:02:41.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:41 smithi016 ceph-mon[31245]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:41 smithi123 ceph-mon[33453]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:41 smithi161 ceph-mon[34061]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-21T02:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:43 smithi016 ceph-mon[31245]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:43 smithi123 ceph-mon[33453]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:44.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:43 smithi161 ceph-mon[34061]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:02:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:46.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:45 smithi016 ceph-mon[31245]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:46.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:45 smithi123 ceph-mon[33453]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:46.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:45 smithi161 ceph-mon[34061]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:47.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.396+0000 7f3c03fff640 1 -- 172.21.15.16:0/1140983708 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3bfc000bc0 con 0x7f3c040b6560 2024-09-21T02:02:47.398 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 msgr2=0x7f3c040b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 0x7f3c040b8a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0058c0 tx=0x7f3c0c005850 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 msgr2=0x7f3c1c19fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400eb10 tx=0x7f3c1400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 shutdown_connections 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c040b6560 0x7f3c040b8a70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1c109e40 0x7f3c1c19fce0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c1c1094f0 0x7f3c1c19aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 --2- 172.21.15.16:0/1140983708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1c103bb0 0x7f3c1c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:47.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 >> 172.21.15.16:0/1140983708 conn(0x7f3c1c0ff600 msgr2=0x7f3c1c101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:02:47.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 shutdown_connections 2024-09-21T02:02:47.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:02:47.399+0000 7f3c01ffb640 1 -- 172.21.15.16:0/1140983708 wait complete. 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:47.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.059 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.9.service 2024-09-21T02:02:48.061 INFO:tasks.cephadm:Deploying osd.10 on smithi123 with /dev/vg_nvme/lv_1... 2024-09-21T02:02:48.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:02:48.318 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:02:48.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:48 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:48.439 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:48 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.439 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:48 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:48 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:48.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:48 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:48 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:48 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:02:48.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:48 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:48.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:48 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:49 smithi123 ceph-mon[33453]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:49 smithi123 ceph-mon[33453]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:02:49.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:49 smithi016 ceph-mon[31245]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:49.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:49 smithi016 ceph-mon[31245]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:02:49.609 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:02:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:02:49.287+0000 7fae7aa7d640 -1 osd.9 0 log_to_monitors true 2024-09-21T02:02:49.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:49 smithi161 ceph-mon[34061]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:49 smithi161 ceph-mon[34061]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-21T02:02:50.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:50 smithi123 ceph-mon[33453]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:02:50.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:50 smithi123 ceph-mon[33453]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:02:50.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:50 smithi123 ceph-mon[33453]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:02:50.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:50 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:50.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:50 smithi123 ceph-mon[33453]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:02:50.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:02:50.402+0000 7fae769ff640 -1 osd.9 0 waiting for initial osdmap 2024-09-21T02:02:50.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:02:50.421+0000 7fae727fd640 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:02:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-mon[31245]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:02:50.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-mon[31245]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:02:50.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-mon[31245]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:02:50.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:50.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:50 smithi016 ceph-mon[31245]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:02:50.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:50 smithi161 ceph-mon[34061]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-21T02:02:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:50 smithi161 ceph-mon[34061]: osdmap e58: 10 total, 9 up, 10 in 2024-09-21T02:02:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:50 smithi161 ceph-mon[34061]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]} : dispatch 2024-09-21T02:02:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:50 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:50.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:50 smithi161 ceph-mon[34061]: from='osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-09-21T02:02:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-21T02:02:51.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi123:vg_nvme/lv_1 2024-09-21T02:02:51.280 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:51.682 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:51.683 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.683 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 -- 172.21.15.123:0/263649184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74102ae0 msgr2=0x7f3c74102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 --2- 172.21.15.123:0/263649184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74102ae0 0x7f3c74102ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009900 tx=0x7f3c6402a410 comp rx=0 tx=0).stop 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 -- 172.21.15.123:0/263649184 shutdown_connections 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 --2- 172.21.15.123:0/263649184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74108d20 0x7f3c7410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 --2- 172.21.15.123:0/263649184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 0x7f3c741087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 --2- 172.21.15.123:0/263649184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74102ae0 0x7f3c74102ef0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:51.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 -- 172.21.15.123:0/263649184 >> 172.21.15.123:0/263649184 conn(0x7f3c74079760 msgr2=0x7f3c74079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 -- 172.21.15.123:0/263649184 shutdown_connections 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.775+0000 7f3c797cf640 1 -- 172.21.15.123:0/263649184 wait complete. 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 Processor -- start 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 -- start start 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74102ae0 0x7f3c7419b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 0x7f3c7419b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c74112840 con 0x7f3c74102ae0 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741126c0 con 0x7f3c74108d20 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c797cf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741129c0 con 0x7f3c741083d0 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c737fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c737fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53992/0 (socket says 172.21.15.123:53992) 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c737fe640 1 -- 172.21.15.123:0/1717790244 learned_addr learned my addr 172.21.15.123:0/1717790244 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-21T02:02:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.776+0000 7f3c72ffd640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74102ae0 0x7f3c7419b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c727fc640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 0x7f3c7419b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 -- 172.21.15.123:0/1717790244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 msgr2=0x7f3c7419b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 0x7f3c7419b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 -- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74102ae0 msgr2=0x7f3c7419b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74102ae0 0x7f3c7419b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 -- 172.21.15.123:0/1717790244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c741a6460 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c737fe640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00eb10 tx=0x7f3c6c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c004b20 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c741a6750 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c6c004cc0 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c010690 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.777+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c741a6c40 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.778+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c74102ef0 con 0x7f3c74108d20 2024-09-21T02:02:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.780+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f3c6c010830 con 0x7f3c74108d20 2024-09-21T02:02:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.780+0000 7f3c6bfff640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 0x7f3c600b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:02:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.780+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6476+0+0 (secure 0 0 0) 0x7f3c6c0e7440 con 0x7f3c74108d20 2024-09-21T02:02:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.780+0000 7f3c72ffd640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 0x7f3c600b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:02:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.781+0000 7f3c72ffd640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 0x7f3c600b8aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c7406c640 tx=0x7f3c640058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:02:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.783+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c6c056710 con 0x7f3c74108d20 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: osdmap e59: 10 total, 9 up, 10 in 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:02:51.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:51.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:02:51.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:02:51.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:02:51.971+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3c74063800 con 0x7f3c600b6590 2024-09-21T02:02:52.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:02:52.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi016 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] boot 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:52.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi016 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] boot 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:02:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:52.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:52.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:52.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi016 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi016 to 4273M 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: osd.9 [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] boot 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: osdmap e60: 10 total, 10 up, 10 in 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:02:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:02:53.685 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:53 smithi123 ceph-mon[33453]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:53.685 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:53 smithi123 ceph-mon[33453]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:53.685 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:53 smithi123 ceph-mon[33453]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T02:02:53.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:53 smithi016 ceph-mon[31245]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:53.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:53 smithi016 ceph-mon[31245]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:53.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:53 smithi016 ceph-mon[31245]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T02:02:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:53 smithi161 ceph-mon[34061]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:53 smithi161 ceph-mon[34061]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-09-21T02:02:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:53 smithi161 ceph-mon[34061]: osdmap e61: 10 total, 10 up, 10 in 2024-09-21T02:02:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/2498474926' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"}]': finished 2024-09-21T02:02:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:02:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:02:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:55 smithi123 ceph-mon[33453]: from='client.? 172.21.15.123:0/1961153486' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/2498474926' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"}]': finished 2024-09-21T02:02:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:02:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:02:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:55 smithi016 ceph-mon[31245]: from='client.? 172.21.15.123:0/1961153486' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/2498474926' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"} : dispatch 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9020569-c79b-468f-a9a0-78a812618945"}]': finished 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: osdmap e62: 11 total, 10 up, 11 in 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:02:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:55 smithi161 ceph-mon[34061]: from='client.? 172.21.15.123:0/1961153486' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:02:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:57 smithi123 ceph-mon[33453]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:57 smithi016 ceph-mon[31245]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:57.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:57 smithi161 ceph-mon[34061]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:59.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:02:59 smithi123 ceph-mon[33453]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:02:59 smithi016 ceph-mon[31245]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:02:59.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:02:59 smithi161 ceph-mon[34061]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:01.704 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:01 smithi123 ceph-mon[33453]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:01 smithi016 ceph-mon[31245]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:01.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:01 smithi161 ceph-mon[34061]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:03.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:03 smithi123 ceph-mon[33453]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:03.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:03:03.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:03.717 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:03 smithi161 ceph-mon[34061]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:03.717 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:03:03.717 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:03 smithi016 ceph-mon[31245]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-21T02:03:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:04.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:04 smithi123 ceph-mon[33453]: Deploying daemon osd.10 on smithi123 2024-09-21T02:03:04.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:04 smithi016 ceph-mon[31245]: Deploying daemon osd.10 on smithi123 2024-09-21T02:03:04.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:04 smithi161 ceph-mon[34061]: Deploying daemon osd.10 on smithi123 2024-09-21T02:03:05.611 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:05 smithi123 ceph-mon[33453]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:05 smithi016 ceph-mon[31245]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:05.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:05 smithi161 ceph-mon[34061]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:07.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:07 smithi123 ceph-mon[33453]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:07 smithi016 ceph-mon[31245]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:07.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:07 smithi161 ceph-mon[34061]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:09.711 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:09 smithi123 ceph-mon[33453]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:09.711 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:09.711 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:09.711 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:09 smithi016 ceph-mon[31245]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:09.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:09 smithi161 ceph-mon[34061]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:09.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:09.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:09.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:11 smithi123 ceph-mon[33453]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:11.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:11 smithi016 ceph-mon[31245]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:11.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:11 smithi161 ceph-mon[34061]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.067+0000 7f3c6bfff640 1 -- 172.21.15.123:0/1717790244 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3c74063800 con 0x7f3c600b6590 2024-09-21T02:03:12.068 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 10 on host 'smithi123' 2024-09-21T02:03:12.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 msgr2=0x7f3c600b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 0x7f3c600b8aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c7406c640 tx=0x7f3c640058d0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 msgr2=0x7f3c7419bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00eb10 tx=0x7f3c6c00c2d0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 shutdown_connections 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3c600b6590 0x7f3c600b8aa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c74108d20 0x7f3c7419bc50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c741083d0 0x7f3c7419b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 --2- 172.21.15.123:0/1717790244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c74102ae0 0x7f3c7419b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 >> 172.21.15.123:0/1717790244 conn(0x7f3c74079760 msgr2=0x7f3c74101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 shutdown_connections 2024-09-21T02:03:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-21T02:03:12.069+0000 7f3c797cf640 1 -- 172.21.15.123:0/1717790244 wait complete. 2024-09-21T02:03:13.074 DEBUG:teuthology.orchestra.run.smithi123:osd.10> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.10.service 2024-09-21T02:03:13.077 INFO:tasks.cephadm:Deploying osd.11 on smithi161 with /dev/vg_nvme/lv_1... 2024-09-21T02:03:13.077 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T02:03:13.334 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:03:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:13 smithi016 ceph-mon[31245]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:13.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:13 smithi161 ceph-mon[34061]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:13.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:13.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:13 smithi123 ceph-mon[33453]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.616 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:14 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.616 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:14 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:14 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:14 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.697 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:14 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:14.697 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:14 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:15.573 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:15 smithi161 ceph-mon[34061]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:15 smithi016 ceph-mon[31245]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:15.769 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:15 smithi123 ceph-mon[33453]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:15.769 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:03:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:03:15.385+0000 7f3ba3183640 -1 osd.10 0 log_to_monitors true 2024-09-21T02:03:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-21T02:03:16.137 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi161:vg_nvme/lv_1 2024-09-21T02:03:16.393 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:03:16.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:16 smithi161 ceph-mon[34061]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:16 smithi161 ceph-mon[34061]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.439 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:16 smithi123 ceph-mon[33453]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.439 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:16 smithi123 ceph-mon[33453]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:16 smithi016 ceph-mon[31245]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:16 smithi016 ceph-mon[31245]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-21T02:03:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 -- 172.21.15.161:0/1459671272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 msgr2=0x7faa24102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/1459671272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa24102e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa20009900 tx=0x7faa2002a410 comp rx=0 tx=0).stop 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 -- 172.21.15.161:0/1459671272 shutdown_connections 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/1459671272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24108d10 0x7faa2410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/1459671272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa241083c0 0x7faa241087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/1459671272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa24102e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 -- 172.21.15.161:0/1459671272 >> 172.21.15.161:0/1459671272 conn(0x7faa24079740 msgr2=0x7faa24079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 -- 172.21.15.161:0/1459671272 shutdown_connections 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.871+0000 7faa2b6b7640 1 -- 172.21.15.161:0/1459671272 wait complete. 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 Processor -- start 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 -- start start 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 0x7faa2419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa24108d10 0x7faa2419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa24113950 con 0x7faa241083c0 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa241137d0 con 0x7faa24108d10 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa2b6b7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa24113ad0 con 0x7faa24102a80 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:16.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58628/0 (socket says 172.21.15.161:58628) 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 -- 172.21.15.161:0/2673106158 learned_addr learned my addr 172.21.15.161:0/2673106158 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa28c52640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 0x7faa2419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 -- 172.21.15.161:0/2673106158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa24108d10 msgr2=0x7faa2419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa24108d10 0x7faa2419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 -- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 msgr2=0x7faa2419a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:16.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 0x7faa2419a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:16.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa29453640 1 -- 172.21.15.161:0/2673106158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241a0330 con 0x7faa24102a80 2024-09-21T02:03:16.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.872+0000 7faa28c52640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 0x7faa2419a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:03:16.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa29453640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa20009a20 tx=0x7faa20004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:16.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2003f030 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa241a05c0 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faa200035f0 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20035070 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.873+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa241a6d80 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.874+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa24103d00 con 0x7faa24102a80 2024-09-21T02:03:16.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.875+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7faa20003aa0 con 0x7faa24102a80 2024-09-21T02:03:16.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.876+0000 7faa1a7fc640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 0x7faa100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:16.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.876+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6613+0+0 (secure 0 0 0) 0x7faa20107d60 con 0x7faa24102a80 2024-09-21T02:03:16.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.876+0000 7faa28c52640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 0x7faa100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:16.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.877+0000 7faa28c52640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 0x7faa100b8a70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faa24079d50 tx=0x7faa14005d30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:16.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:16.879+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faa200cec90 con 0x7faa24102a80 2024-09-21T02:03:17.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:17.069+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7faa24063800 con 0x7faa100b6560 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi123 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi123 to 4272M 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:03:17.605 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:03:17.606 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:17 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi123 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi123 to 4272M 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:03:17.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:17 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:03:17.326+0000 7f3b9f105640 -1 osd.10 0 waiting for initial osdmap 2024-09-21T02:03:17.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:03:17.339+0000 7f3b9af03640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:03:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: osdmap e63: 11 total, 10 up, 11 in 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi123 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi123 to 4272M 2024-09-21T02:03:17.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T02:03:17.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:17 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:18.637 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:18 smithi161 ceph-mon[34061]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:18.637 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:18 smithi161 ceph-mon[34061]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:03:18.637 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:18 smithi161 ceph-mon[34061]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T02:03:18.637 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:18 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:18.637 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:18 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:18.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:18 smithi016 ceph-mon[31245]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:18.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:18 smithi016 ceph-mon[31245]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:03:18.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:18 smithi016 ceph-mon[31245]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T02:03:18.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:18 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:18.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:18 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:18 smithi123 ceph-mon[33453]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:18 smithi123 ceph-mon[33453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-09-21T02:03:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:18 smithi123 ceph-mon[33453]: osdmap e64: 11 total, 10 up, 11 in 2024-09-21T02:03:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:18 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:18 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:03:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:03:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] boot 2024-09-21T02:03:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:03:19.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:19 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] boot 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:03:19.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:19 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:03:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:03:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 894 GiB / 894 GiB avail 2024-09-21T02:03:19.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: osd.10 [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] boot 2024-09-21T02:03:19.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: osdmap e65: 11 total, 11 up, 11 in 2024-09-21T02:03:19.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:19 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/3805246361' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"}]': finished 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:20.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.161:0/318943822' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:03:20.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/3805246361' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"}]': finished 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:20.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.161:0/318943822' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:03:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: osdmap e66: 11 total, 11 up, 11 in 2024-09-21T02:03:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/3805246361' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"} : dispatch 2024-09-21T02:03:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0b0eb6-640a-41fe-b884-b6752e5f75ed"}]': finished 2024-09-21T02:03:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: osdmap e67: 12 total, 11 up, 12 in 2024-09-21T02:03:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.161:0/318943822' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T02:03:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:21 smithi123 ceph-mon[33453]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:21 smithi016 ceph-mon[31245]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:21 smithi161 ceph-mon[34061]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:24.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:23 smithi123 ceph-mon[33453]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-21T02:03:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:23 smithi016 ceph-mon[31245]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-21T02:03:24.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:23 smithi161 ceph-mon[34061]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-21T02:03:25.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:25 smithi161 ceph-mon[34061]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:03:26.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:25 smithi123 ceph-mon[33453]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:03:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:25 smithi016 ceph-mon[31245]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T02:03:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:27 smithi161 ceph-mon[34061]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T02:03:28.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:27 smithi123 ceph-mon[33453]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T02:03:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:27 smithi016 ceph-mon[31245]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T02:03:28.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:28 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:03:28.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:28 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:28 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:03:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:28 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:28 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-21T02:03:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:28 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:29.990 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:29 smithi161 ceph-mon[34061]: Deploying daemon osd.11 on smithi161 2024-09-21T02:03:29.990 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:29 smithi161 ceph-mon[34061]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T02:03:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:29 smithi123 ceph-mon[33453]: Deploying daemon osd.11 on smithi161 2024-09-21T02:03:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:29 smithi123 ceph-mon[33453]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T02:03:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:29 smithi016 ceph-mon[31245]: Deploying daemon osd.11 on smithi161 2024-09-21T02:03:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:29 smithi016 ceph-mon[31245]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-21T02:03:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:31 smithi123 ceph-mon[33453]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T02:03:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:31 smithi016 ceph-mon[31245]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T02:03:32.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:31 smithi161 ceph-mon[34061]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T02:03:33.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:33 smithi161 ceph-mon[34061]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:03:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:33 smithi123 ceph-mon[33453]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:03:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:33 smithi016 ceph-mon[31245]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T02:03:34.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:34.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:34.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:35.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:35 smithi123 ceph-mon[33453]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:35 smithi016 ceph-mon[31245]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:35 smithi161 ceph-mon[34061]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:37.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.577+0000 7faa1a7fc640 1 -- 172.21.15.161:0/2673106158 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7faa24063800 con 0x7faa100b6560 2024-09-21T02:03:37.579 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 11 on host 'smithi161' 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.580+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 msgr2=0x7faa100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.580+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 0x7faa100b8a70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faa24079d50 tx=0x7faa14005d30 comp rx=0 tx=0).stop 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 msgr2=0x7faa2419a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa20009a20 tx=0x7faa20004270 comp rx=0 tx=0).stop 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 shutdown_connections 2024-09-21T02:03:37.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faa100b6560 0x7faa100b8a70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa24108d10 0x7faa2419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa241083c0 0x7faa2419a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 --2- 172.21.15.161:0/2673106158 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa24102a80 0x7faa2419a3c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 >> 172.21.15.161:0/2673106158 conn(0x7faa24079740 msgr2=0x7faa24101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 shutdown_connections 2024-09-21T02:03:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:37.581+0000 7faa2b6b7640 1 -- 172.21.15.161:0/2673106158 wait complete. 2024-09-21T02:03:37.863 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:37.863 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:37.863 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:37.863 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:37.864 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:37.864 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:37.864 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:37.864 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:37.864 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:38.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:38.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:38.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.224 DEBUG:teuthology.orchestra.run.smithi161:osd.11> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.11.service 2024-09-21T02:03:38.227 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-21T02:03:38.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:03:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:38.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:38.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 -- 172.21.15.16:0/516637725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02181025f0 msgr2=0x7f0218102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 --2- 172.21.15.16:0/516637725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02181025f0 0x7f0218102a00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0208009900 tx=0x7f020802a480 comp rx=0 tx=0).stop 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 -- 172.21.15.16:0/516637725 shutdown_connections 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 --2- 172.21.15.16:0/516637725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181101a0 0x7f02181125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 --2- 172.21.15.16:0/516637725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0218102f40 0x7f021810fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 --2- 172.21.15.16:0/516637725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02181025f0 0x7f0218102a00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 -- 172.21.15.16:0/516637725 >> 172.21.15.16:0/516637725 conn(0x7f0218079740 msgr2=0x7f0218079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 -- 172.21.15.16:0/516637725 shutdown_connections 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.008+0000 7f021d168640 1 -- 172.21.15.16:0/516637725 wait complete. 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 Processor -- start 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 -- start start 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 0x7f021819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02181101a0 0x7f02181a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0218115b40 con 0x7f0218102f40 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02181159c0 con 0x7f02181025f0 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021d168640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0218115cc0 con 0x7f02181101a0 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f0216d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 0x7f021819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:39.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42916/0 (socket says 172.21.15.16:42916) 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 -- 172.21.15.16:0/4245702845 learned_addr learned my addr 172.21.15.16:0/4245702845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f021759e640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02181101a0 0x7f02181a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 -- 172.21.15.16:0/4245702845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02181101a0 msgr2=0x7f02181a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02181101a0 0x7f02181a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 -- 172.21.15.16:0/4245702845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 msgr2=0x7f021819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 0x7f021819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f020ffff640 1 -- 172.21.15.16:0/4245702845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02181a24e0 con 0x7f0218102f40 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.009+0000 7f0216d9d640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 0x7f021819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f020ffff640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f020000c970 tx=0x7f020000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200007bb0 con 0x7f0218102f40 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0200007d50 con 0x7f0218102f40 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02181a27d0 con 0x7f0218102f40 2024-09-21T02:03:39.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200004e40 con 0x7f0218102f40 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.010+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02181a8f30 con 0x7f0218102f40 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.011+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021806c240 con 0x7f0218102f40 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.012+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f0200005000 con 0x7f0218102f40 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.014+0000 7f0214d99640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 0x7f02040b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.014+0000 7f0216d9d640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 0x7f02040b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.014+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 7024+0+0 (secure 0 0 0) 0x7f02000e6f40 con 0x7f0218102f40 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.014+0000 7f0216d9d640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 0x7f02040b8a70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f02080057c0 tx=0x7f02080058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:39.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.017+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0200014030 con 0x7f0218102f40 2024-09-21T02:03:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.205+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0218102a00 con 0x7f0218102f40 2024-09-21T02:03:39.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.205+0000 7f0214d99640 1 -- 172.21.15.16:0/4245702845 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v67) ==== 74+0+133 (secure 0 0 0) 0x7f0218102a00 con 0x7f0218102f40 2024-09-21T02:03:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 msgr2=0x7f02040b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 0x7f02040b8a70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f02080057c0 tx=0x7f02080058d0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 msgr2=0x7f021819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f020000c970 tx=0x7f020000ce30 comp rx=0 tx=0).stop 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 shutdown_connections 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f02040b6560 0x7f02040b8a70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02181101a0 0x7f02181a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0218102f40 0x7f021819cb10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 --2- 172.21.15.16:0/4245702845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02181025f0 0x7f021819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:39.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 >> 172.21.15.16:0/4245702845 conn(0x7f0218079740 msgr2=0x7f0218111640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:39.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 shutdown_connections 2024-09-21T02:03:39.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:39.208+0000 7f021d168640 1 -- 172.21.15.16:0/4245702845 wait complete. 2024-09-21T02:03:39.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"num_osds":12,"num_up_osds":11,"osd_up_since":1726884198,"num_in_osds":12,"osd_in_since":1726884199,"num_remapped_pgs":0} 2024-09-21T02:03:40.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:39 smithi123 ceph-mon[33453]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:40.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4245702845' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:40.157 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:39 smithi161 ceph-mon[34061]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:40.157 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4245702845' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:39 smithi016 ceph-mon[31245]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4245702845' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:40.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:03:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:03:40.155+0000 7f8899731640 -1 osd.11 0 log_to_monitors true 2024-09-21T02:03:40.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:03:41.106 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:40 smithi161 ceph-mon[34061]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.106 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:40 smithi161 ceph-mon[34061]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:40 smithi016 ceph-mon[31245]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:40 smithi016 ceph-mon[31245]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.251 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:41.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:40 smithi123 ceph-mon[33453]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:40 smithi123 ceph-mon[33453]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.767+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2812919416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4110200 msgr2=0x7fd2a4112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.767+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2812919416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4110200 0x7fd2a4112620 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009ca0 tx=0x7fd29c02a950 comp rx=0 tx=0).stop 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2812919416 shutdown_connections 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2812919416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4110200 0x7fd2a4112620 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2812919416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4102fa0 0x7fd2a410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2812919416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102650 0x7fd2a4102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2812919416 >> 172.21.15.16:0/2812919416 conn(0x7fd2a4079760 msgr2=0x7fd2a4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2812919416 shutdown_connections 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2812919416 wait complete. 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 Processor -- start 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.768+0000 7fd2aadc5640 1 -- start start 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102fa0 0x7fd2a419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4110200 0x7fd2a41a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4115b70 con 0x7fd2a4102650 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a41159f0 con 0x7fd2a4110200 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2aadc5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4115cf0 con 0x7fd2a4102fa0 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42928/0 (socket says 172.21.15.16:42928) 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 -- 172.21.15.16:0/2842594397 learned_addr learned my addr 172.21.15.16:0/2842594397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a3fff640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102fa0 0x7fd2a419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:41.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a9362640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4110200 0x7fd2a41a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 -- 172.21.15.16:0/2842594397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102fa0 msgr2=0x7fd2a419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102fa0 0x7fd2a419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 -- 172.21.15.16:0/2842594397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4110200 msgr2=0x7fd2a41a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4110200 0x7fd2a41a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 -- 172.21.15.16:0/2842594397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a41a25b0 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.769+0000 7fd2a8b61640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd29800b4a0 tx=0x7fd29800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.770+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd298014070 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.770+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a41a28a0 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.770+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd298010aa0 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.770+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd298017b70 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.770+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41a9000 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.771+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a406c1f0 con 0x7fd2a4102650 2024-09-21T02:03:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.772+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fd298017070 con 0x7fd2a4102650 2024-09-21T02:03:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.773+0000 7fd2a1ffb640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 0x7fd2940b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.773+0000 7fd2a3fff640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 0x7fd2940b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.773+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 7046+0+0 (secure 0 0 0) 0x7fd2980e6380 con 0x7fd2a4102650 2024-09-21T02:03:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.774+0000 7fd2a3fff640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 0x7fd2940b8a70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4104120 tx=0x7fd29000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.776+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd2980b14e0 con 0x7fd2a4102650 2024-09-21T02:03:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.970+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd2a419d770 con 0x7fd2a4102650 2024-09-21T02:03:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.971+0000 7fd2a1ffb640 1 -- 172.21.15.16:0/2842594397 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) ==== 74+0+133 (secure 0 0 0) 0x7fd2a419d770 con 0x7fd2a4102650 2024-09-21T02:03:41.972 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 msgr2=0x7fd2940b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 0x7fd2940b8a70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4104120 tx=0x7fd29000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 msgr2=0x7fd2a419c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd29800b4a0 tx=0x7fd29800b960 comp rx=0 tx=0).stop 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 shutdown_connections 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd2940b6560 0x7fd2940b8a70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2a4110200 0x7fd2a41a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2a4102fa0 0x7fd2a419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 --2- 172.21.15.16:0/2842594397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a4102650 0x7fd2a419c640 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.973+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 >> 172.21.15.16:0/2842594397 conn(0x7fd2a4079760 msgr2=0x7fd2a4108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.974+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 shutdown_connections 2024-09-21T02:03:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:41.974+0000 7fd2aadc5640 1 -- 172.21.15.16:0/2842594397 wait complete. 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:42.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:42.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:42.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:42.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-21T02:03:42.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: osdmap e68: 12 total, 11 up, 12 in 2024-09-21T02:03:42.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:42.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:42.664 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:03:42.215+0000 7f88956b3640 -1 osd.11 0 waiting for initial osdmap 2024-09-21T02:03:42.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:03:42.233+0000 7f888d4a9640 -1 osd.11 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T02:03:42.919 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"num_osds":12,"num_up_osds":11,"osd_up_since":1726884198,"num_in_osds":12,"osd_in_since":1726884199,"num_remapped_pgs":0} 2024-09-21T02:03:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: Detected new or changed devices on smithi161 2024-09-21T02:03:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi161 to 4272M 2024-09-21T02:03:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:03:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T02:03:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2842594397' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:43.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:42 smithi016 ceph-mon[31245]: from='osd.11 ' entity='osd.11' 2024-09-21T02:03:43.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: Detected new or changed devices on smithi161 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi161 to 4272M 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2842594397' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:43.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:42 smithi161 ceph-mon[34061]: from='osd.11 ' entity='osd.11' 2024-09-21T02:03:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: Detected new or changed devices on smithi161 2024-09-21T02:03:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi161 to 4272M 2024-09-21T02:03:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-09-21T02:03:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: osdmap e69: 12 total, 11 up, 12 in 2024-09-21T02:03:43.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:43.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2842594397' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:43.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:42 smithi123 ceph-mon[33453]: from='osd.11 ' entity='osd.11' 2024-09-21T02:03:43.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: purged_snaps scrub starts 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: purged_snaps scrub ok 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] boot 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:03:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: purged_snaps scrub starts 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: purged_snaps scrub ok 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] boot 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:03:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.190 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: purged_snaps scrub starts 2024-09-21T02:03:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: purged_snaps scrub ok 2024-09-21T02:03:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 983 GiB / 983 GiB avail 2024-09-21T02:03:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: osd.11 [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] boot 2024-09-21T02:03:44.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: osdmap e70: 12 total, 12 up, 12 in 2024-09-21T02:03:44.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-21T02:03:44.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 -- 172.21.15.16:0/2130250908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c102a60 msgr2=0x7f3a0c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 --2- 172.21.15.16:0/2130250908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c102a60 0x7f3a0c102e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009900 tx=0x7f39fc02a410 comp rx=0 tx=0).stop 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 -- 172.21.15.16:0/2130250908 shutdown_connections 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 --2- 172.21.15.16:0/2130250908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 0x7f3a0c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 --2- 172.21.15.16:0/2130250908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 --2- 172.21.15.16:0/2130250908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c102a60 0x7f3a0c102e70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 -- 172.21.15.16:0/2130250908 >> 172.21.15.16:0/2130250908 conn(0x7f3a0c079730 msgr2=0x7f3a0c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 -- 172.21.15.16:0/2130250908 shutdown_connections 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.679+0000 7f3a108ed640 1 -- 172.21.15.16:0/2130250908 wait complete. 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 Processor -- start 2024-09-21T02:03:44.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 -- start start 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 0x7f3a0c104480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 0x7f3a0c104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c112810 con 0x7f3a0c1083a0 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c112690 con 0x7f3a0c102a60 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a108ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c112990 con 0x7f3a0c108cf0 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.680+0000 7f3a09d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42936/0 (socket says 172.21.15.16:42936) 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 -- 172.21.15.16:0/4128006204 learned_addr learned my addr 172.21.15.16:0/4128006204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a0ad9d640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 0x7f3a0c104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a0a59c640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 0x7f3a0c104480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 -- 172.21.15.16:0/4128006204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 msgr2=0x7f3a0c104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 0x7f3a0c104f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 -- 172.21.15.16:0/4128006204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 msgr2=0x7f3a0c104480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 0x7f3a0c104480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 -- 172.21.15.16:0/4128006204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c1a86f0 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a0a59c640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 0x7f3a0c104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a09d9b640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a00005fd0 tx=0x7f3a0000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00017070 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0c1a89e0 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a00009c30 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00003c70 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.681+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0c1a8f20 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.682+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0c102e70 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.684+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f3a00003e10 con 0x7f3a0c1083a0 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.684+0000 7f39fb7fe640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 0x7f39f40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:44.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.685+0000 7f3a0a59c640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 0x7f39f40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:44.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.685+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7342+0+0 (secure 0 0 0) 0x7f3a000e73f0 con 0x7f3a0c1083a0 2024-09-21T02:03:44.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.685+0000 7f3a0a59c640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 0x7f39f40b8a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006010 tx=0x7f39fc005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:44.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.688+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3a00020030 con 0x7f3a0c1083a0 2024-09-21T02:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.876+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3a0c1a9210 con 0x7f3a0c1083a0 2024-09-21T02:03:44.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.877+0000 7f39fb7fe640 1 -- 172.21.15.16:0/4128006204 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7f3a000ae050 con 0x7f3a0c1083a0 2024-09-21T02:03:44.878 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 msgr2=0x7f39f40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 0x7f39f40b8a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006010 tx=0x7f39fc005a20 comp rx=0 tx=0).stop 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 msgr2=0x7f3a0c1049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a00005fd0 tx=0x7f3a0000dde0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 shutdown_connections 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f39f40b6560 0x7f39f40b8a70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a0c108cf0 0x7f3a0c104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0c1083a0 0x7f3a0c1049c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 --2- 172.21.15.16:0/4128006204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0c102a60 0x7f3a0c104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 >> 172.21.15.16:0/4128006204 conn(0x7f3a0c079730 msgr2=0x7f3a0c106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 shutdown_connections 2024-09-21T02:03:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:44.879+0000 7f3a108ed640 1 -- 172.21.15.16:0/4128006204 wait complete. 2024-09-21T02:03:45.787 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1726884223,"num_in_osds":12,"osd_in_since":1726884199,"num_remapped_pgs":0} 2024-09-21T02:03:45.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:03:46.050 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:45 smithi016 ceph-mon[31245]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:45 smithi016 ceph-mon[31245]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:03:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:45 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4128006204' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:45 smithi161 ceph-mon[34061]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:45 smithi161 ceph-mon[34061]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:03:46.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:45 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4128006204' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:46.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:45 smithi123 ceph-mon[33453]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:46.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:45 smithi123 ceph-mon[33453]: osdmap e71: 12 total, 12 up, 12 in 2024-09-21T02:03:46.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:45 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4128006204' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T02:03:46.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 -- 172.21.15.16:0/2331043858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4108d10 msgr2=0x7fabf410f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:46.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 --2- 172.21.15.16:0/2331043858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4108d10 0x7fabf410f540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fabec009ca0 tx=0x7fabec02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 -- 172.21.15.16:0/2331043858 shutdown_connections 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 --2- 172.21.15.16:0/2331043858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4108d10 0x7fabf410f540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 --2- 172.21.15.16:0/2331043858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf41087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 --2- 172.21.15.16:0/2331043858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4102a80 0x7fabf4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 -- 172.21.15.16:0/2331043858 >> 172.21.15.16:0/2331043858 conn(0x7fabf4079740 msgr2=0x7fabf4079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 -- 172.21.15.16:0/2331043858 shutdown_connections 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 -- 172.21.15.16:0/2331043858 wait complete. 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.532+0000 7fabf942b640 1 Processor -- start 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 -- start start 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf4199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4108d10 0x7fabf419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabf41128e0 con 0x7fabf4102a80 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fabf4112760 con 0x7fabf4108d10 2024-09-21T02:03:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf942b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fabf4112a60 con 0x7fabf41083c0 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42962/0 (socket says 172.21.15.16:42962) 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 -- 172.21.15.16:0/1758338271 learned_addr learned my addr 172.21.15.16:0/1758338271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf37fe640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4108d10 0x7fabf419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabea7fc640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf4199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 -- 172.21.15.16:0/1758338271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 msgr2=0x7fabf4199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf4199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 -- 172.21.15.16:0/1758338271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4108d10 msgr2=0x7fabf419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4108d10 0x7fabf419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabf2ffd640 1 -- 172.21.15.16:0/1758338271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf419f2e0 con 0x7fabf4102a80 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.533+0000 7fabea7fc640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf4199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf2ffd640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fabe400d950 tx=0x7fabe400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4017070 con 0x7fabf4102a80 2024-09-21T02:03:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf419f5d0 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabe4009c30 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe400f620 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.534+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf41a5d30 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.535+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf4103d00 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.536+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fabe4005030 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.537+0000 7fabf0ff9640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 0x7fabe00b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.537+0000 7fabea7fc640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 0x7fabe00b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.537+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7fabe40e6c30 con 0x7fabf4102a80 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.538+0000 7fabea7fc640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 0x7fabe00b8a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006fd0 tx=0x7fabdc008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.540+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fabe40ad890 con 0x7fabf4102a80 2024-09-21T02:03:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.727+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabf419a4a0 con 0x7fabf4102a80 2024-09-21T02:03:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.728+0000 7fabf0ff9640 1 -- 172.21.15.16:0/1758338271 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18786 (secure 0 0 0) 0x7fabf419a4a0 con 0x7fabf4102a80 2024-09-21T02:03:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:46.730 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:03:44.797807+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 msgr2=0x7fabe00b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 0x7fabe00b8a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006fd0 tx=0x7fabdc008040 comp rx=0 tx=0).stop 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 msgr2=0x7fabf4199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fabe400d950 tx=0x7fabe400de10 comp rx=0 tx=0).stop 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 shutdown_connections 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabe00b6560 0x7fabe00b8a70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabf4108d10 0x7fabf419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabf41083c0 0x7fabf4199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 --2- 172.21.15.16:0/1758338271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf4102a80 0x7fabf4199370 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 >> 172.21.15.16:0/1758338271 conn(0x7fabf4079740 msgr2=0x7fabf4105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 shutdown_connections 2024-09-21T02:03:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:46.731+0000 7fabf942b640 1 -- 172.21.15.16:0/1758338271 wait complete. 2024-09-21T02:03:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:46 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1758338271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:03:47.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:46 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1758338271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:03:47.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:46 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1758338271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:03:47.506 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T01:59:52.388767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T02:03:47.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T02:03:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:48.076 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:47 smithi016 ceph-mon[31245]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:48.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:47 smithi161 ceph-mon[34061]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:48.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:47 smithi123 ceph-mon[33453]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 -- 172.21.15.16:0/523541836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c06b330 msgr2=0x7f379c102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 --2- 172.21.15.16:0/523541836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c06b330 0x7f379c102290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f378c009900 tx=0x7f378c02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 -- 172.21.15.16:0/523541836 shutdown_connections 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 --2- 172.21.15.16:0/523541836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c102be0 0x7f379c108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 --2- 172.21.15.16:0/523541836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 --2- 172.21.15.16:0/523541836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c06b330 0x7f379c102290 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 -- 172.21.15.16:0/523541836 >> 172.21.15.16:0/523541836 conn(0x7f379c079ea0 msgr2=0x7f379c07a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.294+0000 7f37a18e8640 1 -- 172.21.15.16:0/523541836 shutdown_connections 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.295+0000 7f37a18e8640 1 -- 172.21.15.16:0/523541836 wait complete. 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.295+0000 7f37a18e8640 1 Processor -- start 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.295+0000 7f37a18e8640 1 -- start start 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 0x7f379c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f379c112960 con 0x7f379c102be0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f379c1127e0 con 0x7f379c1027d0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f379c112ae0 con 0x7f379c06b330 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60330/0 (socket says 172.21.15.16:60330) 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 -- 172.21.15.16:0/942340612 learned_addr learned my addr 172.21.15.16:0/942340612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379a7fc640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379affd640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 0x7f379c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 -- 172.21.15.16:0/942340612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 msgr2=0x7f379c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 0x7f379c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 -- 172.21.15.16:0/942340612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 msgr2=0x7f379c19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 -- 172.21.15.16:0/942340612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379c1a03f0 con 0x7f379c102be0 2024-09-21T02:03:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379affd640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 0x7f379c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379a7fc640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f379b7fe640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f379400b970 tx=0x7f379400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3794019070 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3794014440 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.296+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f379c1a06e0 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.297+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940075d0 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.297+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f379c1a6f00 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.298+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f379c102290 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.299+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f379401c070 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.300+0000 7f37a08e6640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 0x7f37840b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.300+0000 7f379affd640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 0x7f37840b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.300+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f37940e68e0 con 0x7f379c102be0 2024-09-21T02:03:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.301+0000 7f379affd640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 0x7f37840b8b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f378c006010 tx=0x7f378c000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:48.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.303+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f37940ad540 con 0x7f379c102be0 2024-09-21T02:03:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.492+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f379c02e640 con 0x7f379c102be0 2024-09-21T02:03:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.492+0000 7f37a08e6640 1 -- 172.21.15.16:0/942340612 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7f37940b1920 con 0x7f379c102be0 2024-09-21T02:03:48.493 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-21T02:03:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 msgr2=0x7f37840b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 0x7f37840b8b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f378c006010 tx=0x7f378c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 msgr2=0x7f379c19fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f379400b970 tx=0x7f379400be30 comp rx=0 tx=0).stop 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 shutdown_connections 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f37840b6610 0x7f37840b8b20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f379c102be0 0x7f379c19fd80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f379c1027d0 0x7f379c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 --2- 172.21.15.16:0/942340612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f379c06b330 0x7f379c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 >> 172.21.15.16:0/942340612 conn(0x7f379c079ea0 msgr2=0x7f379c108d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 shutdown_connections 2024-09-21T02:03:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:48.494+0000 7f37a18e8640 1 -- 172.21.15.16:0/942340612 wait complete. 2024-09-21T02:03:49.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:48 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/942340612' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.b on smithi123 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.e on smithi123 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.h on smithi123 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.k on smithi123 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.c on smithi161 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.f on smithi161 2024-09-21T02:03:49.147 INFO:tasks.cephadm:Adding mds.i on smithi161 2024-09-21T02:03:49.148 INFO:tasks.cephadm:Adding mds.l on smithi161 2024-09-21T02:03:49.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l' 2024-09-21T02:03:49.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:48 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/942340612' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:03:49.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:48 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/942340612' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:03:49.410 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:03:49.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.897+0000 7fb55c794640 1 -- 172.21.15.161:0/592539049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554108d30 msgr2=0x7fb55410f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:49.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.897+0000 7fb55c794640 1 --2- 172.21.15.161:0/592539049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554108d30 0x7fb55410f560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009ca0 tx=0x7fb54c02a8e0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 -- 172.21.15.161:0/592539049 shutdown_connections 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 --2- 172.21.15.161:0/592539049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554108d30 0x7fb55410f560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 --2- 172.21.15.161:0/592539049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb5541087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 --2- 172.21.15.161:0/592539049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554102aa0 0x7fb554102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 -- 172.21.15.161:0/592539049 >> 172.21.15.161:0/592539049 conn(0x7fb554079750 msgr2=0x7fb554079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 -- 172.21.15.161:0/592539049 shutdown_connections 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 -- 172.21.15.161:0/592539049 wait complete. 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 Processor -- start 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 -- start start 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.898+0000 7fb55c794640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:49.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb55419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 0x7fb55419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541128e0 con 0x7fb554108d30 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb554112760 con 0x7fb5541083e0 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb554112a60 con 0x7fb554102aa0 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57718/0 (socket says 172.21.15.161:57718) 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 -- 172.21.15.161:0/450787731 learned_addr learned my addr 172.21.15.161:0/450787731 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55ad31640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 0x7fb55419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb559d2f640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb55419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 -- 172.21.15.161:0/450787731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 msgr2=0x7fb55419b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb55419b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 -- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 msgr2=0x7fb55419bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 0x7fb55419bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 -- 172.21.15.161:0/450787731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5541a6480 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb559d2f640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb55419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55ad31640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 0x7fb55419bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55a530640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d950 tx=0x7fb55000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550017070 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb550009c30 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5541a6770 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55000f620 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.899+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5541a6d30 con 0x7fb554102aa0 2024-09-21T02:03:49.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.901+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb554102eb0 con 0x7fb554102aa0 2024-09-21T02:03:49.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.902+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fb550005030 con 0x7fb554102aa0 2024-09-21T02:03:49.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.903+0000 7fb54b7fe640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 0x7fb5400b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:49.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.903+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7fb5500e6c30 con 0x7fb554102aa0 2024-09-21T02:03:49.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.904+0000 7fb559d2f640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 0x7fb5400b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:49.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.904+0000 7fb559d2f640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 0x7fb5400b8aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:49.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:49.906+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb5500ad890 con 0x7fb554102aa0 2024-09-21T02:03:50.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.095+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}) -- 0x7fb554063800 con 0x7fb5400b6590 2024-09-21T02:03:50.096 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:49 smithi161 ceph-mon[34061]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:50.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.103+0000 7fb54b7fe640 1 -- 172.21.15.161:0/450787731 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fb554063800 con 0x7fb5400b6590 2024-09-21T02:03:50.104 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mds.all update... 2024-09-21T02:03:50.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 msgr2=0x7fb5400b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 0x7fb5400b8aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544008040 comp rx=0 tx=0).stop 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 msgr2=0x7fb55419b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d950 tx=0x7fb55000de10 comp rx=0 tx=0).stop 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 shutdown_connections 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb5400b6590 0x7fb5400b8aa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:50.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb554108d30 0x7fb55419bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5541083e0 0x7fb55419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 --2- 172.21.15.161:0/450787731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb554102aa0 0x7fb55419b1f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 >> 172.21.15.161:0/450787731 conn(0x7fb554079750 msgr2=0x7fb554106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 shutdown_connections 2024-09-21T02:03:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-21T02:03:50.106+0000 7fb55c794640 1 -- 172.21.15.161:0/450787731 wait complete. 2024-09-21T02:03:50.123 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:49 smithi123 ceph-mon[33453]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:49 smithi016 ceph-mon[31245]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:50.839 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.a.service 2024-09-21T02:03:50.841 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.d.service 2024-09-21T02:03:50.884 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.g.service 2024-09-21T02:03:50.927 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.j.service 2024-09-21T02:03:50.970 DEBUG:teuthology.orchestra.run.smithi123:mds.b> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.b.service 2024-09-21T02:03:50.973 DEBUG:teuthology.orchestra.run.smithi123:mds.e> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.e.service 2024-09-21T02:03:51.016 DEBUG:teuthology.orchestra.run.smithi123:mds.h> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.h.service 2024-09-21T02:03:51.059 DEBUG:teuthology.orchestra.run.smithi123:mds.k> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.k.service 2024-09-21T02:03:51.101 DEBUG:teuthology.orchestra.run.smithi161:mds.c> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.c.service 2024-09-21T02:03:51.104 DEBUG:teuthology.orchestra.run.smithi161:mds.f> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.f.service 2024-09-21T02:03:51.146 DEBUG:teuthology.orchestra.run.smithi161:mds.i> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.i.service 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:51 smithi016 ceph-mon[31245]: Deploying daemon mds.l on smithi161 2024-09-21T02:03:51.189 DEBUG:teuthology.orchestra.run.smithi161:mds.l> sudo journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.l.service 2024-09-21T02:03:51.232 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-21T02:03:51.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:03:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-09-21T02:03:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:51 smithi123 ceph-mon[33453]: Deploying daemon mds.l on smithi161 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:51.369 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:51.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:51.370 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:51 smithi161 ceph-mon[34061]: Deploying daemon mds.l on smithi161 2024-09-21T02:03:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.978+0000 7fe980d5f640 1 -- 172.21.15.16:0/3476443419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 msgr2=0x7fe97c10f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.978+0000 7fe980d5f640 1 --2- 172.21.15.16:0/3476443419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c10f5a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe974009ca0 tx=0x7fe97402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 -- 172.21.15.16:0/3476443419 shutdown_connections 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 --2- 172.21.15.16:0/3476443419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c10f5a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 --2- 172.21.15.16:0/3476443419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 0x7fe97c108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 --2- 172.21.15.16:0/3476443419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 -- 172.21.15.16:0/3476443419 >> 172.21.15.16:0/3476443419 conn(0x7fe97c079770 msgr2=0x7fe97c079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 -- 172.21.15.16:0/3476443419 shutdown_connections 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.979+0000 7fe980d5f640 1 -- 172.21.15.16:0/3476443419 wait complete. 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 Processor -- start 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 -- start start 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 0x7fe97c19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c112930 con 0x7fe97c108d70 2024-09-21T02:03:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c1127b0 con 0x7fe97c108420 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe980d5f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c112ab0 con 0x7fe97c102ae0 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe97ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe97a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe97a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:59290/0 (socket says 172.21.15.16:59290) 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe97ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60350/0 (socket says 172.21.15.16:60350) 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe97a59c640 1 -- 172.21.15.16:0/799332212 learned_addr learned my addr 172.21.15.16:0/799332212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.980+0000 7fe979d9b640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 0x7fe97c19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 -- 172.21.15.16:0/799332212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 msgr2=0x7fe97c19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 0x7fe97c19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 -- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 msgr2=0x7fe97c19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 -- 172.21.15.16:0/799332212 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c1a02a0 con 0x7fe97c102ae0 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97ad9d640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:03:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe97a59c640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00d770 tx=0x7fe96c00db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c017070 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c1a0590 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe96c009820 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c004b20 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.981+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c1a6db0 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.982+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97c103d60 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.984+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fe96c0099d0 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.985+0000 7fe96b7fe640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 0x7fe9640b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.985+0000 7fe979d9b640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 0x7fe9640b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.985+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7fe96c0e6db0 con 0x7fe97c102ae0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.985+0000 7fe979d9b640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 0x7fe9640b8aa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe97c106420 tx=0x7fe97000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:52.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:51.987+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe96c0ad970 con 0x7fe97c102ae0 2024-09-21T02:03:52.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.213+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe97c102ef0 con 0x7fe97c102ae0 2024-09-21T02:03:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.214+0000 7fe96b7fe640 1 -- 172.21.15.16:0/799332212 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fe96c0b1d50 con 0x7fe97c102ae0 2024-09-21T02:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"btime":"2024-09-21T01:56:00:153260+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-21T02:03:52.216 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-09-21T02:03:52.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 msgr2=0x7fe9640b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:52.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 0x7fe9640b8aa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe97c106420 tx=0x7fe97000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 msgr2=0x7fe97c19a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00d770 tx=0x7fe96c00db70 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 shutdown_connections 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe9640b6590 0x7fe9640b8aa0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe97c108d70 0x7fe97c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97c108420 0x7fe97c19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 --2- 172.21.15.16:0/799332212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe97c102ae0 0x7fe97c19a3f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 >> 172.21.15.16:0/799332212 conn(0x7fe97c079770 msgr2=0x7fe97c101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 shutdown_connections 2024-09-21T02:03:52.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:52.216+0000 7fe980d5f640 1 -- 172.21.15.16:0/799332212 wait complete. 2024-09-21T02:03:52.665 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:52 smithi161 systemd[1]: Starting Ceph mds.l for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:03:52.979 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:52 smithi161 podman[63352]: 2024-09-21 02:03:52.611288797 +0000 UTC m=+0.015965033 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:03:52.979 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:52 smithi161 podman[63352]: 2024-09-21 02:03:52.734471254 +0000 UTC m=+0.139147477 container create 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T02:03:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:53 smithi016 ceph-mon[31245]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:53 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/799332212' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:53 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6832/2556025136,v1:172.21.15.161:6833/2556025136] up:boot 2024-09-21T02:03:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:53 smithi016 ceph-mon[31245]: fsmap 1 up:standby 2024-09-21T02:03:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:03:53.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mon[34061]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:53.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/799332212' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6832/2556025136,v1:172.21.15.161:6833/2556025136] up:boot 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mon[34061]: fsmap 1 up:standby 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:52 smithi161 podman[63352]: 2024-09-21 02:03:52.978853001 +0000 UTC m=+0.383529236 container init 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:52 smithi161 podman[63352]: 2024-09-21 02:03:52.981820811 +0000 UTC m=+0.386497042 container start 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=) 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-mds[63372]: -- 172.21.15.161:0/2966050996 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa7bf17860 con 0x55aa800bfa80 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: starting mds.l at 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:53 smithi161 bash[63352]: 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a 2024-09-21T02:03:53.415 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:03:53 smithi161 systemd[1]: Started Ceph mds.l for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:03:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:53 smithi123 ceph-mon[33453]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:53 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/799332212' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:53 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6832/2556025136,v1:172.21.15.161:6833/2556025136] up:boot 2024-09-21T02:03:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:53 smithi123 ceph-mon[33453]: fsmap 1 up:standby 2024-09-21T02:03:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:54 smithi016 ceph-mon[31245]: Deploying daemon mds.k on smithi123 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:54.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:54 smithi161 ceph-mon[34061]: Deploying daemon mds.k on smithi123 2024-09-21T02:03:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:54.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:54.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:54 smithi123 ceph-mon[33453]: Deploying daemon mds.k on smithi123 2024-09-21T02:03:54.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:03:55.150 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 systemd[1]: Starting Ceph mds.k for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:03:55.267 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:03:55.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:55 smithi016 ceph-mon[31245]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:55.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:55 smithi161 ceph-mon[34061]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:55.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:55 smithi123 ceph-mon[33453]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:55.483 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 podman[61511]: 2024-09-21 02:03:55.122307681 +0000 UTC m=+0.015783455 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:03:55.483 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 podman[61511]: 2024-09-21 02:03:55.238633187 +0000 UTC m=+0.132108946 container create 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T02:03:55.484 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 podman[61511]: 2024-09-21 02:03:55.483115595 +0000 UTC m=+0.376591366 container init 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:03:55.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 podman[61511]: 2024-09-21 02:03:55.486237789 +0000 UTC m=+0.379713549 container start 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T02:03:55.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 ceph-mds[61530]: -- 172.21.15.123:0/1078502630 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562308ab9860 con 0x56230cc61600 2024-09-21T02:03:55.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: starting mds.k at 2024-09-21T02:03:55.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 bash[61511]: 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 2024-09-21T02:03:55.771 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:03:55 smithi123 systemd[1]: Started Ceph mds.k for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 -- 172.21.15.16:0/3319202536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00102a60 msgr2=0x7fab00102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 --2- 172.21.15.16:0/3319202536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00102a60 0x7fab00102e70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faae8009900 tx=0x7faae802a410 comp rx=0 tx=0).stop 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 -- 172.21.15.16:0/3319202536 shutdown_connections 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 --2- 172.21.15.16:0/3319202536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab00108cf0 0x7fab0010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 --2- 172.21.15.16:0/3319202536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab001083a0 0x7fab001087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 --2- 172.21.15.16:0/3319202536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00102a60 0x7fab00102e70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 -- 172.21.15.16:0/3319202536 >> 172.21.15.16:0/3319202536 conn(0x7fab00079730 msgr2=0x7fab00079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 -- 172.21.15.16:0/3319202536 shutdown_connections 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.781+0000 7fab05e26640 1 -- 172.21.15.16:0/3319202536 wait complete. 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 Processor -- start 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 -- start start 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab00102a60 0x7fab0019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 0x7fab0019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab00112810 con 0x7fab00108cf0 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fab00112690 con 0x7fab001083a0 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7fab05e26640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab00112990 con 0x7fab00102a60 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7faafffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7faafffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60360/0 (socket says 172.21.15.16:60360) 2024-09-21T02:03:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7faafffff640 1 -- 172.21.15.16:0/3435992733 learned_addr learned my addr 172.21.15.16:0/3435992733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7faaff7fe640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab00102a60 0x7fab0019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.782+0000 7faafeffd640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 0x7fab0019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 -- 172.21.15.16:0/3435992733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab00102a60 msgr2=0x7fab0019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab00102a60 0x7fab0019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 -- 172.21.15.16:0/3435992733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 msgr2=0x7fab0019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 0x7fab0019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 -- 172.21.15.16:0/3435992733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab001a02c0 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafeffd640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 0x7fab0019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafffff640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faaf800eb10 tx=0x7faaf800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8004b20 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faaf8004cc0 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8010690 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab001a05b0 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.783+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab001a6d10 con 0x7fab00108cf0 2024-09-21T02:03:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.784+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab00102e70 con 0x7fab00108cf0 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.786+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7faaf8005020 con 0x7fab00108cf0 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.786+0000 7faafcff9640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 0x7faaec0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.787+0000 7faaff7fe640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 0x7faaec0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.787+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7faaf80e75b0 con 0x7fab00108cf0 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.787+0000 7faaff7fe640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 0x7faaec0b8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faae80057c0 tx=0x7faae80058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:03:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:55.791+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faaf80ae210 con 0x7fab00108cf0 2024-09-21T02:03:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.070+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab001a6f40 con 0x7fab00108cf0 2024-09-21T02:03:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.070+0000 7faafcff9640 1 -- 172.21.15.16:0/3435992733 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7faaf80b25f0 con 0x7fab00108cf0 2024-09-21T02:03:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:03:56.073 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"btime":"2024-09-21T02:03:53:020427+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-21T02:03:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 msgr2=0x7faaec0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 0x7faaec0b8af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faae80057c0 tx=0x7faae80058d0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 msgr2=0x7fab0019fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faaf800eb10 tx=0x7faaf800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 shutdown_connections 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7faaec0b65e0 0x7faaec0b8af0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab00108cf0 0x7fab0019fb90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab001083a0 0x7fab0019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 --2- 172.21.15.16:0/3435992733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab00102a60 0x7fab0019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:03:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 >> 172.21.15.16:0/3435992733 conn(0x7fab00079730 msgr2=0x7fab00101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:03:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 shutdown_connections 2024-09-21T02:03:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:03:56.073+0000 7fab05e26640 1 -- 172.21.15.16:0/3435992733 wait complete. 2024-09-21T02:03:56.904 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.904 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.904 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.904 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: Deploying daemon mds.j on smithi016 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3435992733' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6832/3641506877,v1:172.21.15.123:6833/3641506877] up:boot 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: fsmap 2 up:standby 2024-09-21T02:03:56.905 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:56 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: Deploying daemon mds.j on smithi016 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3435992733' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6832/3641506877,v1:172.21.15.123:6833/3641506877] up:boot 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: fsmap 2 up:standby 2024-09-21T02:03:56.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:56 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:03:56.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: Deploying daemon mds.j on smithi016 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3435992733' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6832/3641506877,v1:172.21.15.123:6833/3641506877] up:boot 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: fsmap 2 up:standby 2024-09-21T02:03:56.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:56 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:03:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:57 smithi016 ceph-mon[31245]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:57.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:57 smithi161 ceph-mon[34061]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:58.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:57 smithi123 ceph-mon[33453]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:58.641 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 systemd[1]: Starting Ceph mds.j for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:03:58.908 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 podman[72975]: 2024-09-21 02:03:58.543873231 +0000 UTC m=+0.016207830 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:03:58.909 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 podman[72975]: 2024-09-21 02:03:58.689373945 +0000 UTC m=+0.161708598 container create d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux ) 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 podman[72975]: 2024-09-21 02:03:58.940502831 +0000 UTC m=+0.412837430 container init d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 podman[72975]: 2024-09-21 02:03:58.943542309 +0000 UTC m=+0.415876905 container start d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 ceph-mds[72994]: -- 172.21.15.16:0/3828612992 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56514d0a9860 con 0x565151251180 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j[72990]: starting mds.j at 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:59 smithi016 bash[72975]: d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 2024-09-21T02:03:59.408 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:03:59 smithi016 systemd[1]: Started Ceph mds.j for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:03:59.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:59.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:59.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:03:59 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:03:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:03:59 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:00.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:03:59 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:00.697 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:00.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:00 smithi016 ceph-mon[31245]: Deploying daemon mds.d on smithi016 2024-09-21T02:04:00.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:00 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6834/3026745530,v1:172.21.15.16:6835/3026745530] up:boot 2024-09-21T02:04:00.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:00 smithi016 ceph-mon[31245]: fsmap 3 up:standby 2024-09-21T02:04:00.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:00 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:04:00.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:00 smithi161 ceph-mon[34061]: Deploying daemon mds.d on smithi016 2024-09-21T02:04:00.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:00 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6834/3026745530,v1:172.21.15.16:6835/3026745530] up:boot 2024-09-21T02:04:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:00 smithi161 ceph-mon[34061]: fsmap 3 up:standby 2024-09-21T02:04:00.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:00 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:04:01.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:00 smithi123 ceph-mon[33453]: Deploying daemon mds.d on smithi016 2024-09-21T02:04:01.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:00 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6834/3026745530,v1:172.21.15.16:6835/3026745530] up:boot 2024-09-21T02:04:01.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:00 smithi123 ceph-mon[33453]: fsmap 3 up:standby 2024-09-21T02:04:01.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:00 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "j"} : dispatch 2024-09-21T02:04:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 -- 172.21.15.16:0/626585557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 msgr2=0x7f52241087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 --2- 172.21.15.16:0/626585557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241087f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5218009900 tx=0x7f521802a410 comp rx=0 tx=0).stop 2024-09-21T02:04:01.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 -- 172.21.15.16:0/626585557 shutdown_connections 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 --2- 172.21.15.16:0/626585557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224108d30 0x7f522410f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 --2- 172.21.15.16:0/626585557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241087f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 --2- 172.21.15.16:0/626585557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224102aa0 0x7f5224102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 -- 172.21.15.16:0/626585557 >> 172.21.15.16:0/626585557 conn(0x7f5224079750 msgr2=0x7f5224079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 -- 172.21.15.16:0/626585557 shutdown_connections 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 -- 172.21.15.16:0/626585557 wait complete. 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.373+0000 7f5229cce640 1 Processor -- start 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 -- start start 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224102aa0 0x7f5224199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224108d30 0x7f522419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f52241128a0 con 0x7f52241083e0 2024-09-21T02:04:01.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5224112720 con 0x7f5224108d30 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5229cce640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5224112a20 con 0x7f5224102aa0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58296/0 (socket says 172.21.15.16:58296) 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 -- 172.21.15.16:0/3591640754 learned_addr learned my addr 172.21.15.16:0/3591640754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f52237fe640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224102aa0 0x7f5224199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5223fff640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224108d30 0x7f522419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 -- 172.21.15.16:0/3591640754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224102aa0 msgr2=0x7f5224199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224102aa0 0x7f5224199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 -- 172.21.15.16:0/3591640754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224108d30 msgr2=0x7f522419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224108d30 0x7f522419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.374+0000 7f5222ffd640 1 -- 172.21.15.16:0/3591640754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522419f280 con 0x7f52241083e0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5222ffd640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5218004770 tx=0x7f5218004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:01.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521803f030 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522419f510 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5218004440 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218035070 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.375+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52241a5cd0 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.376+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5224103d20 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.378+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f5218042030 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.379+0000 7f5220ff9640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 0x7f520c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.379+0000 7f52237fe640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 0x7f520c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.379+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f5218107780 con 0x7f52241083e0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.380+0000 7f52237fe640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 0x7f520c0b8a70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f521400bbe0 tx=0x7f521400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.382+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f52180ce3e0 con 0x7f52241083e0 2024-09-21T02:04:01.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.622+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5224102eb0 con 0x7f52241083e0 2024-09-21T02:04:01.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.622+0000 7f5220ff9640 1 -- 172.21.15.16:0/3591640754 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3091 (secure 0 0 0) 0x7f52180d27c0 con 0x7f52241083e0 2024-09-21T02:04:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:01.624 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"btime":"2024-09-21T02:03:59:649772+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-21T02:04:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 msgr2=0x7f520c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 0x7f520c0b8a70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f521400bbe0 tx=0x7f521400b390 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 msgr2=0x7f52241998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5218004770 tx=0x7f5218004040 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 shutdown_connections 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f520c0b6560 0x7f520c0b8a70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5224108d30 0x7f522419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52241083e0 0x7f52241998b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 --2- 172.21.15.16:0/3591640754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5224102aa0 0x7f5224199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 >> 172.21.15.16:0/3591640754 conn(0x7f5224079750 msgr2=0x7f5224106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 shutdown_connections 2024-09-21T02:04:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:01.625+0000 7f5229cce640 1 -- 172.21.15.16:0/3591640754 wait complete. 2024-09-21T02:04:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:01 smithi016 ceph-mon[31245]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:01 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3591640754' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:01.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:01 smithi161 ceph-mon[34061]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:01.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:01 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3591640754' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:02.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:01 smithi123 ceph-mon[33453]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:02.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:01 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3591640754' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:02.908 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:04:02 smithi016 systemd[1]: Started Ceph mds.d for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:04.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6836/3790973256,v1:172.21.15.16:6837/3790973256] up:boot 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: fsmap 4 up:standby 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:04.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:03 smithi123 ceph-mon[33453]: Deploying daemon mds.e on smithi123 2024-09-21T02:04:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6836/3790973256,v1:172.21.15.16:6837/3790973256] up:boot 2024-09-21T02:04:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: fsmap 4 up:standby 2024-09-21T02:04:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:04:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:04.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:03 smithi016 ceph-mon[31245]: Deploying daemon mds.e on smithi123 2024-09-21T02:04:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6836/3790973256,v1:172.21.15.16:6837/3790973256] up:boot 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: fsmap 4 up:standby 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "d"} : dispatch 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:03 smithi161 ceph-mon[34061]: Deploying daemon mds.e on smithi123 2024-09-21T02:04:04.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:04.998 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:04 smithi123 systemd[1]: Starting Ceph mds.e for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:04:05.270 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:04 smithi123 podman[61788]: 2024-09-21 02:04:04.900261315 +0000 UTC m=+0.015999004 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:04:05.270 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 podman[61788]: 2024-09-21 02:04:05.018726115 +0000 UTC m=+0.134463788 container create fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-21T02:04:05.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- 172.21.15.16:0/3479265436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 msgr2=0x7f0368102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 --2- 172.21.15.16:0/3479265436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f0368102640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0364009900 tx=0x7f036402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- 172.21.15.16:0/3479265436 shutdown_connections 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 --2- 172.21.15.16:0/3479265436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03681084c0 0x7f036810ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 --2- 172.21.15.16:0/3479265436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0368107b70 0x7f0368107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 --2- 172.21.15.16:0/3479265436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f0368102640 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- 172.21.15.16:0/3479265436 >> 172.21.15.16:0/3479265436 conn(0x7f0368079e40 msgr2=0x7f036807a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- 172.21.15.16:0/3479265436 shutdown_connections 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- 172.21.15.16:0/3479265436 wait complete. 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 Processor -- start 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.294+0000 7f036f4c2640 1 -- start start 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368107b70 0x7f036819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03681084c0 0x7f03681a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03681117b0 con 0x7f0368102230 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0368111630 con 0x7f0368107b70 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036f4c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0368111930 con 0x7f03681084c0 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58318/0 (socket says 172.21.15.16:58318) 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 -- 172.21.15.16:0/2989797355 learned_addr learned my addr 172.21.15.16:0/2989797355 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036ca5d640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368107b70 0x7f036819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036da5f640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03681084c0 0x7f03681a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 -- 172.21.15.16:0/2989797355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03681084c0 msgr2=0x7f03681a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03681084c0 0x7f03681a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 -- 172.21.15.16:0/2989797355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368107b70 msgr2=0x7f036819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368107b70 0x7f036819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 -- 172.21.15.16:0/2989797355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03681a26a0 con 0x7f0368102230 2024-09-21T02:04:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.295+0000 7f036d25e640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0364009230 tx=0x7f0364004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.296+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036403f030 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.296+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0364003a80 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.296+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03681a2930 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.296+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364035070 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.296+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03681a90f0 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.297+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0368102640 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.298+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f0364042030 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.299+0000 7f035a7fc640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 0x7f03540b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.299+0000 7f036ca5d640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 0x7f03540b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.300+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f03641076d0 con 0x7f0368102230 2024-09-21T02:04:05.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.300+0000 7f036ca5d640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 0x7f03540b8a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0368107720 tx=0x7f035c010040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:05.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.304+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f03640d2670 con 0x7f0368102230 2024-09-21T02:04:05.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.533+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03681a92c0 con 0x7f0368102230 2024-09-21T02:04:05.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.533+0000 7f035a7fc640 1 -- 172.21.15.16:0/2989797355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7f03640d75b0 con 0x7f0368102230 2024-09-21T02:04:05.534 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:05.535 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"btime":"2024-09-21T02:04:02:713815+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-21T02:04:05.535 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 msgr2=0x7f03540b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 0x7f03540b8a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0368107720 tx=0x7f035c010040 comp rx=0 tx=0).stop 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 msgr2=0x7f036819c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0364009230 tx=0x7f0364004270 comp rx=0 tx=0).stop 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 shutdown_connections 2024-09-21T02:04:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f03540b6560 0x7f03540b8a70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03681084c0 0x7f03681a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368107b70 0x7f036819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 --2- 172.21.15.16:0/2989797355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0368102230 0x7f036819c730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 >> 172.21.15.16:0/2989797355 conn(0x7f0368079e40 msgr2=0x7f0368105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 shutdown_connections 2024-09-21T02:04:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:05.536+0000 7f036f4c2640 1 -- 172.21.15.16:0/2989797355 wait complete. 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 podman[61788]: 2024-09-21 02:04:05.296437034 +0000 UTC m=+0.412174705 container init fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 podman[61788]: 2024-09-21 02:04:05.29928148 +0000 UTC m=+0.415019151 container start fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mds[61807]: -- 172.21.15.123:0/1830806879 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56029edaf860 con 0x5602a2f57600 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: starting mds.e at 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 bash[61788]: fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 2024-09-21T02:04:05.728 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:04:05 smithi123 systemd[1]: Started Ceph mds.e for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:05.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:05.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:05.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:05 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2989797355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:06.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:06.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:05 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2989797355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:06.062 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:06.062 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.062 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.063 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:06.063 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:06.063 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:06.063 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:06.063 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:05 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2989797355' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:06.993 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:06 smithi161 ceph-mon[34061]: Deploying daemon mds.f on smithi161 2024-09-21T02:04:06.993 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:06 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:boot 2024-09-21T02:04:06.993 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:06 smithi161 ceph-mon[34061]: fsmap 5 up:standby 2024-09-21T02:04:06.993 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:06 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:04:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:06 smithi123 ceph-mon[33453]: Deploying daemon mds.f on smithi161 2024-09-21T02:04:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:06 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:boot 2024-09-21T02:04:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:06 smithi123 ceph-mon[33453]: fsmap 5 up:standby 2024-09-21T02:04:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:06 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:04:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:06 smithi016 ceph-mon[31245]: Deploying daemon mds.f on smithi161 2024-09-21T02:04:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:06 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:boot 2024-09-21T02:04:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:06 smithi016 ceph-mon[31245]: fsmap 5 up:standby 2024-09-21T02:04:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:06 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:04:07.665 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 podman[63636]: 2024-09-21 02:04:07.604484989 +0000 UTC m=+0.143676660 container create a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T02:04:08.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:07 smithi123 ceph-mon[33453]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:07 smithi016 ceph-mon[31245]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:08.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:07 smithi161 ceph-mon[34061]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 podman[63636]: 2024-09-21 02:04:07.857620233 +0000 UTC m=+0.396811904 container init a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 podman[63636]: 2024-09-21 02:04:07.860677091 +0000 UTC m=+0.399868764 container start a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 ceph-mds[63655]: -- 172.21.15.161:0/3962933546 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651214df860 con 0x565125687a80 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: starting mds.f at 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 bash[63636]: a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 2024-09-21T02:04:08.165 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:04:07 smithi161 systemd[1]: Started Ceph mds.f for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:08.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:08.485 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.944+0000 7f6f962c0640 1 -- 172.21.15.16:0/424647486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f901083d0 msgr2=0x7f6f901087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.944+0000 7f6f962c0640 1 --2- 172.21.15.16:0/424647486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f901083d0 0x7f6f901087e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009900 tx=0x7f6f8402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 -- 172.21.15.16:0/424647486 shutdown_connections 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 --2- 172.21.15.16:0/424647486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90108d20 0x7f6f9010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 --2- 172.21.15.16:0/424647486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f901083d0 0x7f6f901087e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 --2- 172.21.15.16:0/424647486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f90102ae0 0x7f6f90102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 -- 172.21.15.16:0/424647486 >> 172.21.15.16:0/424647486 conn(0x7f6f900797b0 msgr2=0x7f6f90079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 -- 172.21.15.16:0/424647486 shutdown_connections 2024-09-21T02:04:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 -- 172.21.15.16:0/424647486 wait complete. 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 Processor -- start 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.945+0000 7f6f962c0640 1 -- start start 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90102ae0 0x7f6f900ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 0x7f6f900ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90112890 con 0x7f6f90108d20 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90112710 con 0x7f6f901083d0 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f962c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90112a10 con 0x7f6f90102ae0 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47760/0 (socket says 172.21.15.16:47760) 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 -- 172.21.15.16:0/2848436529 learned_addr learned my addr 172.21.15.16:0/2848436529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f8f7fe640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 0x7f6f900ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f8ffff640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90102ae0 0x7f6f900ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 -- 172.21.15.16:0/2848436529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90102ae0 msgr2=0x7f6f900ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90102ae0 0x7f6f900ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 -- 172.21.15.16:0/2848436529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 msgr2=0x7f6f900ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 0x7f6f900ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f9485d640 1 -- 172.21.15.16:0/2848436529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f901a8770 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.946+0000 7f6f8f7fe640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 0x7f6f900ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f9485d640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009fd0 tx=0x7f6f8800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88017070 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f901a8a60 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f88003e00 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88010460 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.947+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f901a8f20 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.948+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f90102ef0 con 0x7f6f90108d20 2024-09-21T02:04:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.950+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f6f88003970 con 0x7f6f90108d20 2024-09-21T02:04:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.950+0000 7f6f8d7fa640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 0x7f6f780b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.951+0000 7f6f8ffff640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 0x7f6f780b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.951+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f6f880e6fd0 con 0x7f6f90108d20 2024-09-21T02:04:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.951+0000 7f6f8ffff640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 0x7f6f780b8a70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6f901010c0 tx=0x7f6f80005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:08.958+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6f880b2010 con 0x7f6f90108d20 2024-09-21T02:04:09.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:09.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: Deploying daemon mds.c on smithi161 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6834/1279942273,v1:172.21.15.161:6835/1279942273] up:boot 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: fsmap 6 up:standby 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:09 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: Deploying daemon mds.c on smithi161 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6834/1279942273,v1:172.21.15.161:6835/1279942273] up:boot 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: fsmap 6 up:standby 2024-09-21T02:04:09.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:09 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:04:09.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.188+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f90100970 con 0x7f6f90108d20 2024-09-21T02:04:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.188+0000 7f6f8d7fa640 1 -- 172.21.15.16:0/2848436529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5572 (secure 0 0 0) 0x7f6f90100970 con 0x7f6f90108d20 2024-09-21T02:04:09.189 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"btime":"2024-09-21T02:04:08:735137+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-21T02:04:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2024-09-21T02:04:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.190+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 msgr2=0x7f6f780b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.190+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 0x7f6f780b8a70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6f901010c0 tx=0x7f6f80005f90 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.190+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 msgr2=0x7f6f901000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.190+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009fd0 tx=0x7f6f8800b260 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 shutdown_connections 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f780b6560 0x7f6f780b8a70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f90108d20 0x7f6f901000f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f901083d0 0x7f6f900ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 --2- 172.21.15.16:0/2848436529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f90102ae0 0x7f6f900ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 >> 172.21.15.16:0/2848436529 conn(0x7f6f900797b0 msgr2=0x7f6f90101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 shutdown_connections 2024-09-21T02:04:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:09.191+0000 7f6f962c0640 1 -- 172.21.15.16:0/2848436529 wait complete. 2024-09-21T02:04:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: Deploying daemon mds.c on smithi161 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6834/1279942273,v1:172.21.15.161:6835/1279942273] up:boot 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: fsmap 6 up:standby 2024-09-21T02:04:09.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:09 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:04:10.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:10 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2848436529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:10.165 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 systemd[1]: Starting Ceph mds.c for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:04:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:10 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2848436529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:10 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2848436529' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:10.538 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 podman[63906]: 2024-09-21 02:04:10.156791696 +0000 UTC m=+0.016013851 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:04:10.538 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 podman[63906]: 2024-09-21 02:04:10.277160424 +0000 UTC m=+0.136382575 container create 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 podman[63906]: 2024-09-21 02:04:10.538381401 +0000 UTC m=+0.397603564 container init 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 podman[63906]: 2024-09-21 02:04:10.541393662 +0000 UTC m=+0.400615824 container start 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 ceph-mds[63927]: -- 172.21.15.161:0/272232690 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55acbf82d860 con 0x55acc39d5600 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: starting mds.c at 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 bash[63906]: 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 2024-09-21T02:04:10.915 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:04:10 smithi161 systemd[1]: Started Ceph mds.c for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: Deploying daemon mds.h on smithi123 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:boot 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: fsmap 7 up:standby 2024-09-21T02:04:12.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:11 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:04:12.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: Deploying daemon mds.h on smithi123 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:boot 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: fsmap 7 up:standby 2024-09-21T02:04:12.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:11 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:04:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: Deploying daemon mds.h on smithi123 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:boot 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: fsmap 7 up:standby 2024-09-21T02:04:12.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:11 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:04:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:12.770 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:12 smithi123 systemd[1]: Starting Ceph mds.h for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:04:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- 172.21.15.16:0/1463415744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 msgr2=0x7f62d0102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 --2- 172.21.15.16:0/1463415744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d0102ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009900 tx=0x7f62c002a410 comp rx=0 tx=0).stop 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- 172.21.15.16:0/1463415744 shutdown_connections 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 --2- 172.21.15.16:0/1463415744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d0108d20 0x7f62d010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 --2- 172.21.15.16:0/1463415744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d01083d0 0x7f62d01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 --2- 172.21.15.16:0/1463415744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d0102ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- 172.21.15.16:0/1463415744 >> 172.21.15.16:0/1463415744 conn(0x7f62d0079760 msgr2=0x7f62d0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- 172.21.15.16:0/1463415744 shutdown_connections 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- 172.21.15.16:0/1463415744 wait complete. 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 Processor -- start 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.825+0000 7f62d71d1640 1 -- start start 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 0x7f62d019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 0x7f62d019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0112840 con 0x7f62d0102ae0 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01126c0 con 0x7f62d01083d0 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d71d1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01129c0 con 0x7f62d0108d20 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47778/0 (socket says 172.21.15.16:47778) 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 -- 172.21.15.16:0/109134647 learned_addr learned my addr 172.21.15.16:0/109134647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62cffff640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 0x7f62d019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d576e640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 0x7f62d019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 -- 172.21.15.16:0/109134647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 msgr2=0x7f62d019bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 0x7f62d019bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 -- 172.21.15.16:0/109134647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 msgr2=0x7f62d019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 0x7f62d019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 -- 172.21.15.16:0/109134647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d01a6440 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62cffff640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 0x7f62d019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d576e640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 0x7f62d019bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.826+0000 7f62d4f6d640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009230 tx=0x7f62c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.827+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c003f030 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.827+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d01a66d0 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.827+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f62c0003a80 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.827+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0035070 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.827+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01a6bb0 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.828+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d0103d10 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.829+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f62c0042030 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.830+0000 7f62cdffb640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 0x7f62b80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.830+0000 7f62cffff640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 0x7f62b80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.830+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f62c0107a60 con 0x7f62d0102ae0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.831+0000 7f62cffff640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 0x7f62b80b8aa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009fd0 tx=0x7f62c4009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:12.834+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f62c00ce6c0 con 0x7f62d0102ae0 2024-09-21T02:04:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.060+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d0102ef0 con 0x7f62d0102ae0 2024-09-21T02:04:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.060+0000 7f62cdffb640 1 -- 172.21.15.16:0/109134647 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6400 (secure 0 0 0) 0x7f62c00d5e20 con 0x7f62d0102ae0 2024-09-21T02:04:13.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:13.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"btime":"2024-09-21T02:04:11:013232+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-21T02:04:13.062 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 msgr2=0x7f62b80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 0x7f62b80b8aa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009fd0 tx=0x7f62c4009680 comp rx=0 tx=0).stop 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 msgr2=0x7f62d019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009230 tx=0x7f62c0004270 comp rx=0 tx=0).stop 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 shutdown_connections 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f62b80b6590 0x7f62b80b8aa0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:13.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62d0108d20 0x7f62d019bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:13.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62d01083d0 0x7f62d019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:13.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 --2- 172.21.15.16:0/109134647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62d0102ae0 0x7f62d019b1b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:13.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 >> 172.21.15.16:0/109134647 conn(0x7f62d0079760 msgr2=0x7f62d0101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:13.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 shutdown_connections 2024-09-21T02:04:13.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:13.063+0000 7f62d71d1640 1 -- 172.21.15.16:0/109134647 wait complete. 2024-09-21T02:04:13.168 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:12 smithi123 podman[62062]: 2024-09-21 02:04:12.814039969 +0000 UTC m=+0.015871546 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:04:13.169 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:12 smithi123 podman[62062]: 2024-09-21 02:04:12.915339404 +0000 UTC m=+0.117171031 container create 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T02:04:13.169 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 podman[62062]: 2024-09-21 02:04:13.16825028 +0000 UTC m=+0.370081866 container init 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T02:04:13.520 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 podman[62062]: 2024-09-21 02:04:13.171162806 +0000 UTC m=+0.372994385 container start 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:04:13.520 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mds[62082]: -- 172.21.15.123:0/1268142005 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555957447860 con 0x55595b5efa80 2024-09-21T02:04:13.520 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: starting mds.h at 2024-09-21T02:04:13.521 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 bash[62062]: 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 2024-09-21T02:04:13.521 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:04:13 smithi123 systemd[1]: Started Ceph mds.h for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/109134647' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:13.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:13.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:13 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:13.966 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/109134647' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:13.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:13 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/109134647' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:14.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:14.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:14.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:14.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:13 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:14 smithi123 ceph-mon[33453]: Deploying daemon mds.i on smithi161 2024-09-21T02:04:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:14 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:boot 2024-09-21T02:04:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:14 smithi123 ceph-mon[33453]: fsmap 8 up:standby 2024-09-21T02:04:15.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:14 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:04:15.065 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:14 smithi161 ceph-mon[34061]: Deploying daemon mds.i on smithi161 2024-09-21T02:04:15.065 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:14 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:boot 2024-09-21T02:04:15.065 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:14 smithi161 ceph-mon[34061]: fsmap 8 up:standby 2024-09-21T02:04:15.065 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:14 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:04:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:14 smithi016 ceph-mon[31245]: Deploying daemon mds.i on smithi161 2024-09-21T02:04:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:14 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:boot 2024-09-21T02:04:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:14 smithi016 ceph-mon[31245]: fsmap 8 up:standby 2024-09-21T02:04:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:14 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:04:15.415 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 systemd[1]: Starting Ceph mds.i for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:04:15.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:15.751 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:15 smithi161 ceph-mon[34061]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:15.751 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 podman[64174]: 2024-09-21 02:04:15.370055668 +0000 UTC m=+0.016332298 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:04:15.751 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 podman[64174]: 2024-09-21 02:04:15.473754962 +0000 UTC m=+0.120031581 container create 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T02:04:15.752 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 podman[64174]: 2024-09-21 02:04:15.751457296 +0000 UTC m=+0.397733928 container init 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T02:04:15.997 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:15 smithi123 ceph-mon[33453]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:16.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:15 smithi016 ceph-mon[31245]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:16.165 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 podman[64174]: 2024-09-21 02:04:15.755170572 +0000 UTC m=+0.401447192 container start 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux ) 2024-09-21T02:04:16.165 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 ceph-mds[64193]: -- 172.21.15.161:0/807455472 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef31533860 con 0x55ef356db600 2024-09-21T02:04:16.165 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: starting mds.i at 2024-09-21T02:04:16.165 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 bash[64174]: 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b 2024-09-21T02:04:16.165 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:04:15 smithi161 systemd[1]: Started Ceph mds.i for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:16.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 -- 172.21.15.16:0/489287050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 msgr2=0x7f5be0102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 --2- 172.21.15.16:0/489287050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0102e90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009900 tx=0x7f5bd002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 -- 172.21.15.16:0/489287050 shutdown_connections 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 --2- 172.21.15.16:0/489287050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be010f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 --2- 172.21.15.16:0/489287050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 0x7f5be01087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 --2- 172.21.15.16:0/489287050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0102e90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 -- 172.21.15.16:0/489287050 >> 172.21.15.16:0/489287050 conn(0x7f5be0079740 msgr2=0x7f5be0079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 -- 172.21.15.16:0/489287050 shutdown_connections 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.493+0000 7f5be6e35640 1 -- 172.21.15.16:0/489287050 wait complete. 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 Processor -- start 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 -- start start 2024-09-21T02:04:16.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 0x7f5be01997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0112830 con 0x7f5be0102a80 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01126b0 con 0x7f5be0108d10 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be6e35640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01129b0 con 0x7f5be01083c0 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be4bd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be4bd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47788/0 (socket says 172.21.15.16:47788) 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.494+0000 7f5be4bd1640 1 -- 172.21.15.16:0/2755394871 learned_addr learned my addr 172.21.15.16:0/2755394871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5bdffff640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 0x7f5be01997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be53d2640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 -- 172.21.15.16:0/2755394871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 msgr2=0x7f5be01997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 0x7f5be01997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 -- 172.21.15.16:0/2755394871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 msgr2=0x7f5be019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 -- 172.21.15.16:0/2755394871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be019f0b0 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be53d2640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be4bd1640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009a20 tx=0x7f5bd0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd003f030 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be019f340 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5bd00035f0 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0035070 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.495+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be019f820 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.497+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be0103d00 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.498+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f5bd0003aa0 con 0x7f5be0102a80 2024-09-21T02:04:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.499+0000 7f5bddffb640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 0x7f5bc80b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.499+0000 7f5bdffff640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 0x7f5bc80b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.499+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f5bd0107e80 con 0x7f5be0102a80 2024-09-21T02:04:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.499+0000 7f5bdffff640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 0x7f5bc80b8aa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5be0100d40 tx=0x7f5bd400a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.502+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5bd00ceae0 con 0x7f5be0102a80 2024-09-21T02:04:16.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.733+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5be0102e90 con 0x7f5be0102a80 2024-09-21T02:04:16.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.734+0000 7f5bddffb640 1 -- 172.21.15.16:0/2755394871 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8058 (secure 0 0 0) 0x7f5bd00b5090 con 0x7f5be0102a80 2024-09-21T02:04:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:16.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"btime":"2024-09-21T02:04:16:703162+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24667,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24683,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-21T02:04:16.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-09-21T02:04:16.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 msgr2=0x7f5bc80b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:16.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 0x7f5bc80b8aa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5be0100d40 tx=0x7f5bd400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 msgr2=0x7f5be0199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009a20 tx=0x7f5bd0004270 comp rx=0 tx=0).stop 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 shutdown_connections 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5bc80b6590 0x7f5bc80b8aa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5be0108d10 0x7f5be019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be01083c0 0x7f5be01997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 --2- 172.21.15.16:0/2755394871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5be0102a80 0x7f5be0199200 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.736+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 >> 172.21.15.16:0/2755394871 conn(0x7f5be0079740 msgr2=0x7f5be0101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.737+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 shutdown_connections 2024-09-21T02:04:16.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:16.737+0000 7f5be6e35640 1 -- 172.21.15.16:0/2755394871 wait complete. 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: Deploying daemon mds.g on smithi016 2024-09-21T02:04:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:17.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:boot 2024-09-21T02:04:17.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: fsmap 9 up:standby 2024-09-21T02:04:17.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:04:17.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:16 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2755394871' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:17.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: Deploying daemon mds.g on smithi016 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:boot 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: fsmap 9 up:standby 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:04:17.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:16 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2755394871' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:17.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: Deploying daemon mds.g on smithi016 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:boot 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: fsmap 9 up:standby 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:04:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:16 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2755394871' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:19 smithi016 ceph-mon[31245]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:19.159 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 podman[74104]: 2024-09-21 02:04:19.10314326 +0000 UTC m=+0.147929201 container create d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T02:04:19.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:19 smithi161 ceph-mon[34061]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:19.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:19 smithi123 ceph-mon[33453]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:19.658 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 podman[74104]: 2024-09-21 02:04:19.348939804 +0000 UTC m=+0.393725746 container init d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T02:04:19.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 podman[74104]: 2024-09-21 02:04:19.351809348 +0000 UTC m=+0.396595290 container start d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:04:19.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 ceph-mds[74123]: -- 172.21.15.16:0/1969578680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56473435b860 con 0x564738503600 2024-09-21T02:04:19.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: starting mds.g at 2024-09-21T02:04:19.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 bash[74104]: d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a 2024-09-21T02:04:19.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:04:19 smithi016 systemd[1]: Started Ceph mds.g for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:19.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: Deploying daemon mds.a on smithi016 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6838/1623043260,v1:172.21.15.16:6839/1623043260] up:boot 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: fsmap 10 up:standby 2024-09-21T02:04:20.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:20 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:04:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: Deploying daemon mds.a on smithi016 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6838/1623043260,v1:172.21.15.16:6839/1623043260] up:boot 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: fsmap 10 up:standby 2024-09-21T02:04:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:20 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: Deploying daemon mds.a on smithi016 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6838/1623043260,v1:172.21.15.16:6839/1623043260] up:boot 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: fsmap 10 up:standby 2024-09-21T02:04:20.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:20 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:04:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 -- 172.21.15.16:0/593681592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c10de40 msgr2=0x7f497c110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 --2- 172.21.15.16:0/593681592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c10de40 0x7f497c110310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f497402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 -- 172.21.15.16:0/593681592 shutdown_connections 2024-09-21T02:04:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 --2- 172.21.15.16:0/593681592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c10de40 0x7f497c110310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 --2- 172.21.15.16:0/593681592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c1052c0 0x7f497c10d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 --2- 172.21.15.16:0/593681592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c104890 0x7f497c104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 -- 172.21.15.16:0/593681592 >> 172.21.15.16:0/593681592 conn(0x7f497c079730 msgr2=0x7f497c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 -- 172.21.15.16:0/593681592 shutdown_connections 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.553+0000 7f4984692640 1 -- 172.21.15.16:0/593681592 wait complete. 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 Processor -- start 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 -- start start 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c1052c0 0x7f497c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c10de40 0x7f497c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113980 con 0x7f497c104890 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113800 con 0x7f497c10de40 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f4984692640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113b00 con 0x7f497c1052c0 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f498242e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f498242e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42954/0 (socket says 172.21.15.16:42954) 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.554+0000 7f498242e640 1 -- 172.21.15.16:0/2824334013 learned_addr learned my addr 172.21.15.16:0/2824334013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f4982c2f640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c10de40 0x7f497c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f4981c2d640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c1052c0 0x7f497c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 -- 172.21.15.16:0/2824334013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c1052c0 msgr2=0x7f497c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c1052c0 0x7f497c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 -- 172.21.15.16:0/2824334013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c10de40 msgr2=0x7f497c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c10de40 0x7f497c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 -- 172.21.15.16:0/2824334013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c1a02e0 con 0x7f497c104890 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f498242e640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4978009d20 tx=0x7f497800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.555+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978010630 con 0x7f497c104890 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.556+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c1a05d0 con 0x7f497c104890 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.556+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4978010c60 con 0x7f497c104890 2024-09-21T02:04:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.556+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978018c10 con 0x7f497c104890 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.556+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c102680 con 0x7f497c104890 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.557+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c06c140 con 0x7f497c104890 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.559+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f497801f020 con 0x7f497c104890 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.560+0000 7f49737fe640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 0x7f49680b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.560+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f49780e74f0 con 0x7f497c104890 2024-09-21T02:04:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.561+0000 7f4981c2d640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 0x7f49680b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.561+0000 7f4981c2d640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 0x7f49680b8af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f497c102350 tx=0x7f496c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.566+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f49780ae0b0 con 0x7f497c104890 2024-09-21T02:04:21.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:21 smithi016 ceph-mon[31245]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.809+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f497c104ca0 con 0x7f497c104890 2024-09-21T02:04:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.810+0000 7f49737fe640 1 -- 172.21.15.16:0/2824334013 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8884 (secure 0 0 0) 0x7f49780b2490 con 0x7f497c104890 2024-09-21T02:04:21.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:21.811 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"btime":"2024-09-21T02:04:20:060803+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14841,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24667,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24683,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-21T02:04:21.811 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-09-21T02:04:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 msgr2=0x7f49680b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 0x7f49680b8af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f497c102350 tx=0x7f496c005f90 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 msgr2=0x7f497c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4978009d20 tx=0x7f497800c6a0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 shutdown_connections 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.812+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f49680b65e0 0x7f49680b8af0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f497c10de40 0x7f497c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f497c1052c0 0x7f497c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 --2- 172.21.15.16:0/2824334013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f497c104890 0x7f497c19a370 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 >> 172.21.15.16:0/2824334013 conn(0x7f497c079730 msgr2=0x7f497c1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 shutdown_connections 2024-09-21T02:04:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:21.813+0000 7f4984692640 1 -- 172.21.15.16:0/2824334013 wait complete. 2024-09-21T02:04:21.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:21 smithi161 ceph-mon[34061]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:22.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:21 smithi123 ceph-mon[33453]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:22.809 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:22 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2824334013' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:22.810 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:04:22 smithi016 podman[74467]: 2024-09-21 02:04:22.773333033 +0000 UTC m=+1.130043309 container start 5cceb684e14078da0d95e15a17d7a715063ccbce961c4491ff853799f9111f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:04:22.810 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:04:22 smithi016 ceph-mds[74503]: -- 172.21.15.16:0/3676425617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592640ef860 con 0x559268297180 2024-09-21T02:04:22.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:22 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2824334013' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:23.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:22 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2824334013' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:23.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:04:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: starting mds.a at 2024-09-21T02:04:23.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:04:22 smithi016 bash[74467]: 5cceb684e14078da0d95e15a17d7a715063ccbce961c4491ff853799f9111f9f 2024-09-21T02:04:23.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:04:22 smithi016 systemd[1]: Started Ceph mds.a for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:23.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:23.597 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:23 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:23 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T02:04:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:23 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:24.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:24.764 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:24 smithi123 ceph-mon[33453]: Deploying daemon mds.b on smithi123 2024-09-21T02:04:24.764 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:24 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6840/3323661058,v1:172.21.15.16:6841/3323661058] up:boot 2024-09-21T02:04:24.764 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:24 smithi123 ceph-mon[33453]: fsmap 11 up:standby 2024-09-21T02:04:24.765 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:24 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:04:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:24 smithi016 ceph-mon[31245]: Deploying daemon mds.b on smithi123 2024-09-21T02:04:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:24 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6840/3323661058,v1:172.21.15.16:6841/3323661058] up:boot 2024-09-21T02:04:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:24 smithi016 ceph-mon[31245]: fsmap 11 up:standby 2024-09-21T02:04:24.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:24 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:04:24.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:24 smithi161 ceph-mon[34061]: Deploying daemon mds.b on smithi123 2024-09-21T02:04:24.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:24 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6840/3323661058,v1:172.21.15.16:6841/3323661058] up:boot 2024-09-21T02:04:24.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:24 smithi161 ceph-mon[34061]: fsmap 11 up:standby 2024-09-21T02:04:24.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:24 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:04:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:25.020 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:24 smithi123 systemd[1]: Starting Ceph mds.b for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:04:25.414 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 podman[62331]: 2024-09-21 02:04:25.050902787 +0000 UTC m=+0.015755455 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:04:25.414 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 podman[62331]: 2024-09-21 02:04:25.169999419 +0000 UTC m=+0.134852080 container create 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:04:25.414 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 podman[62331]: 2024-09-21 02:04:25.414251034 +0000 UTC m=+0.379103693 container init 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 -- 172.21.15.16:0/2049783562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd4102ae0 msgr2=0x7fafd4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 --2- 172.21.15.16:0/2049783562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd4102ae0 0x7fafd4102ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009900 tx=0x7fafc402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 -- 172.21.15.16:0/2049783562 shutdown_connections 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 --2- 172.21.15.16:0/2049783562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4108d20 0x7fafd410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 --2- 172.21.15.16:0/2049783562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd41083d0 0x7fafd41087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 --2- 172.21.15.16:0/2049783562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd4102ae0 0x7fafd4102ef0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 -- 172.21.15.16:0/2049783562 >> 172.21.15.16:0/2049783562 conn(0x7fafd4079760 msgr2=0x7fafd4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 -- 172.21.15.16:0/2049783562 shutdown_connections 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.485+0000 7fafda54b640 1 -- 172.21.15.16:0/2049783562 wait complete. 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 Processor -- start 2024-09-21T02:04:25.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 -- start start 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4102ae0 0x7fafd419b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4108d20 0x7fafd419bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafd4112840 con 0x7fafd41083d0 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41126c0 con 0x7fafd4108d20 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafda54b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fafd41129c0 con 0x7fafd4102ae0 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafd3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4102ae0 0x7fafd419b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafd37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42996/0 (socket says 172.21.15.16:42996) 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.486+0000 7fafd37fe640 1 -- 172.21.15.16:0/513333734 learned_addr learned my addr 172.21.15.16:0/513333734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd8ae8640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4108d20 0x7fafd419bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 -- 172.21.15.16:0/513333734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4102ae0 msgr2=0x7fafd419b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:25.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4102ae0 0x7fafd419b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 -- 172.21.15.16:0/513333734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4108d20 msgr2=0x7fafd419bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4108d20 0x7fafd419bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 -- 172.21.15.16:0/513333734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd41a6350 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd37fe640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c970 tx=0x7fafc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8014070 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafc8004440 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd41a6640 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8003d40 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.487+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd41a6bd0 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.488+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd4103d10 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.490+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fafc8017070 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.491+0000 7fafd17fa640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 0x7fafbc0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.491+0000 7fafd3fff640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 0x7fafbc0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.491+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7fafc80e7190 con 0x7fafd41083d0 2024-09-21T02:04:25.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.491+0000 7fafd3fff640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 0x7fafbc0b8b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fafd406c6c0 tx=0x7fafc40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:25.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.494+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fafc801e030 con 0x7fafd41083d0 2024-09-21T02:04:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:25 smithi016 ceph-mon[31245]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:25.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.738+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd4102ef0 con 0x7fafd41083d0 2024-09-21T02:04:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.739+0000 7fafd17fa640 1 -- 172.21.15.16:0/513333734 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10539 (secure 0 0 0) 0x7fafc80ade70 con 0x7fafd41083d0 2024-09-21T02:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"btime":"2024-09-21T02:04:25:545252+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14841,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14853,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24667,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24683,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-21T02:04:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-09-21T02:04:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.742+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 msgr2=0x7fafbc0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.742+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 0x7fafbc0b8b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fafd406c6c0 tx=0x7fafc40058d0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.742+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 msgr2=0x7fafd419b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.742+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c970 tx=0x7fafc800ce30 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 shutdown_connections 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafbc0b6610 0x7fafbc0b8b20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafd4108d20 0x7fafd419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafd41083d0 0x7fafd419b690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 --2- 172.21.15.16:0/513333734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd4102ae0 0x7fafd419b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 >> 172.21.15.16:0/513333734 conn(0x7fafd4079760 msgr2=0x7fafd4101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 shutdown_connections 2024-09-21T02:04:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:25.743+0000 7fafda54b640 1 -- 172.21.15.16:0/513333734 wait complete. 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:25 smithi123 ceph-mon[33453]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 podman[62331]: 2024-09-21 02:04:25.41710238 +0000 UTC m=+0.381955040 container start 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 ceph-mds[62350]: -- 172.21.15.123:0/1842065830 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56041fd15860 con 0x560423ebd600 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: starting mds.b at 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 bash[62331]: 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 2024-09-21T02:04:25.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:25 smithi123 systemd[1]: Started Ceph mds.b for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:04:25.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:25 smithi161 ceph-mon[34061]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:26.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph --format=json-pretty osd lspools 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:boot 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: fsmap 12 up:standby 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:26.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/513333734' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:26.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:boot 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: fsmap 12 up:standby 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:26.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/513333734' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:26.975 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:boot 2024-09-21T02:04:26.975 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: fsmap 12 up:standby 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:26.976 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/513333734' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:27.413 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:27.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:27.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.817 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 -- 172.21.15.16:0/343054422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824108d20 msgr2=0x7ff82410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 --2- 172.21.15.16:0/343054422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824108d20 0x7ff82410f550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009ca0 tx=0x7ff81c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 -- 172.21.15.16:0/343054422 shutdown_connections 2024-09-21T02:04:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 --2- 172.21.15.16:0/343054422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824108d20 0x7ff82410f550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 --2- 172.21.15.16:0/343054422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff8241087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 --2- 172.21.15.16:0/343054422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824102ae0 0x7ff824102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 -- 172.21.15.16:0/343054422 >> 172.21.15.16:0/343054422 conn(0x7ff824079760 msgr2=0x7ff824079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 -- 172.21.15.16:0/343054422 shutdown_connections 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 -- 172.21.15.16:0/343054422 wait complete. 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.907+0000 7ff82ac92640 1 Processor -- start 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 -- start start 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff82419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 0x7ff82419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff824112890 con 0x7ff824102ae0 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff824112710 con 0x7ff8241083d0 2024-09-21T02:04:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82ac92640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff824112a10 con 0x7ff824108d20 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44060/0 (socket says 172.21.15.16:44060) 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 -- 172.21.15.16:0/1046396598 learned_addr learned my addr 172.21.15.16:0/1046396598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff823fff640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff82419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82922f640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 0x7ff82419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 -- 172.21.15.16:0/1046396598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 msgr2=0x7ff82419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 0x7ff82419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 -- 172.21.15.16:0/1046396598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 msgr2=0x7ff82419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff82419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 -- 172.21.15.16:0/1046396598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8241a0320 con 0x7ff824102ae0 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff82922f640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 0x7ff82419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff823fff640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff82419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:04:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.908+0000 7ff828a2e640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff81800d770 tx=0x7ff81800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.909+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818017070 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.909+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8180054b0 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.909+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8241a0610 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.909+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818004c60 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.909+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8241a6d70 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.910+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff824103d10 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.911+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff818004e00 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.912+0000 7ff821ffb640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 0x7ff8140b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.913+0000 7ff823fff640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 0x7ff8140b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.913+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7ff8180e6a10 con 0x7ff824102ae0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.913+0000 7ff823fff640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 0x7ff8140b8a70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff82406cba0 tx=0x7ff810005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:27.917+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff8180ad670 con 0x7ff824102ae0 2024-09-21T02:04:27.944 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:27.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:27.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.117+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff824102ef0 con 0x7ff824102ae0 2024-09-21T02:04:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.117+0000 7ff821ffb640 1 -- 172.21.15.16:0/1046396598 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) ==== 84+0+65 (secure 0 0 0) 0x7ff8180b1a50 con 0x7ff824102ae0 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:04:28.119 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 msgr2=0x7ff8140b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 0x7ff8140b8a70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff82406cba0 tx=0x7ff810005f90 comp rx=0 tx=0).stop 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 msgr2=0x7ff82419a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff81800d770 tx=0x7ff81800dc30 comp rx=0 tx=0).stop 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 shutdown_connections 2024-09-21T02:04:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff8140b6560 0x7ff8140b8a70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff824108d20 0x7ff82419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8241083d0 0x7ff82419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 --2- 172.21.15.16:0/1046396598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff824102ae0 0x7ff82419a3b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.121+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 >> 172.21.15.16:0/1046396598 conn(0x7ff824079760 msgr2=0x7ff824101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.122+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 shutdown_connections 2024-09-21T02:04:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:28.122+0000 7ff82ac92640 1 -- 172.21.15.16:0/1046396598 wait complete. 2024-09-21T02:04:28.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1046396598' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:04:28.835 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-21T02:04:28.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-21T02:04:28.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1046396598' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:04:29.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1046396598' entity='client.admin' cmd={"prefix": "osd lspools", "format": "json-pretty"} : dispatch 2024-09-21T02:04:29.104 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:29.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 -- 172.21.15.16:0/2645426766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 msgr2=0x7fabcc10d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/2645426766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc10d860 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009900 tx=0x7fabc002a410 comp rx=0 tx=0).stop 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 -- 172.21.15.16:0/2645426766 shutdown_connections 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/2645426766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc10de20 0x7fabcc110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/2645426766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc10d860 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/2645426766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc1049c0 0x7fabcc104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 -- 172.21.15.16:0/2645426766 >> 172.21.15.16:0/2645426766 conn(0x7fabcc079710 msgr2=0x7fabcc079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 -- 172.21.15.16:0/2645426766 shutdown_connections 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.613+0000 7fabd1a9a640 1 -- 172.21.15.16:0/2645426766 wait complete. 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.614+0000 7fabd1a9a640 1 Processor -- start 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.614+0000 7fabd1a9a640 1 -- start start 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1049c0 0x7fabcc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:29.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc10de20 0x7fabcc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc113930 con 0x7fabcc105310 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc1137b0 con 0x7fabcc1049c0 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabd1a9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc113ab0 con 0x7fabcc10de20 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44076/0 (socket says 172.21.15.16:44076) 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 -- 172.21.15.16:0/3935252102 learned_addr learned my addr 172.21.15.16:0/3935252102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcbfff640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc10de20 0x7fabcc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcb7fe640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1049c0 0x7fabcc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 -- 172.21.15.16:0/3935252102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc10de20 msgr2=0x7fabcc19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc10de20 0x7fabcc19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 -- 172.21.15.16:0/3935252102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1049c0 msgr2=0x7fabcc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1049c0 0x7fabcc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:29.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.615+0000 7fabcaffd640 1 -- 172.21.15.16:0/3935252102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc1a0350 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabcaffd640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fabc0005940 tx=0x7fabc00040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc003f030 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fabc0003a80 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc1a05e0 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0035070 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.616+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc1a6da0 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.617+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc06c280 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.618+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fabc0042030 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.619+0000 7fabc8ff9640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 0x7fabb40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.620+0000 7fabcb7fe640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 0x7fabb40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.620+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7fabc01073f0 con 0x7fabcc105310 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.620+0000 7fabcb7fe640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 0x7fabb40b8af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fabbc005fd0 tx=0x7fabbc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:29.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.624+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fabc00d2430 con 0x7fabcc105310 2024-09-21T02:04:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:29.843+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fabcc108f20 con 0x7fabcc105310 2024-09-21T02:04:29.844 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:29.844 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.844 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:29.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:29.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:29 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:29.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:29 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:30.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:30.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:29 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:04:30.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.602+0000 7fabc8ff9640 1 -- 172.21.15.16:0/3935252102 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) ==== 138+0+0 (secure 0 0 0) 0x7fabc00d7370 con 0x7fabcc105310 2024-09-21T02:04:30.603 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 msgr2=0x7fabb40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 0x7fabb40b8af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fabbc005fd0 tx=0x7fabbc00b390 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 msgr2=0x7fabcc19a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fabc0005940 tx=0x7fabc00040e0 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 shutdown_connections 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fabb40b65e0 0x7fabb40b8af0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabcc10de20 0x7fabcc19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabcc105310 0x7fabcc19a980 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 --2- 172.21.15.16:0/3935252102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabcc1049c0 0x7fabcc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:30.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.606+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 >> 172.21.15.16:0/3935252102 conn(0x7fabcc079710 msgr2=0x7fabcc1008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:30.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.607+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 shutdown_connections 2024-09-21T02:04:30.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:30.607+0000 7fabd1a9a640 1 -- 172.21.15.16:0/3935252102 wait complete. 2024-09-21T02:04:30.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:30 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:04:30.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:30 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:30 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:04:30.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:30.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:30 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:30 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} : dispatch 2024-09-21T02:04:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:30 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-21T02:04:31.563 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:31.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:31 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:31.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:31 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3935252102' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: osdmap e72: 12 total, 12 up, 12 in 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:32.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:04:32.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:04:32.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:31 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2742818378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04102a80 msgr2=0x7f9f04102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2742818378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04102a80 0x7f9f04102e90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009900 tx=0x7f9ef402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2742818378 shutdown_connections 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2742818378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f04108d10 0x7f9f0410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2742818378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f041083c0 0x7f9f041087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2742818378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04102a80 0x7f9f04102e90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2742818378 >> 172.21.15.16:0/2742818378 conn(0x7f9f04079740 msgr2=0x7f9f04079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2742818378 shutdown_connections 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.092+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2742818378 wait complete. 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 Processor -- start 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 -- start start 2024-09-21T02:04:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f04102a80 0x7f9f0419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f041083c0 0x7f9f0419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04112890 con 0x7f9f04108d10 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04112710 con 0x7f9f04102a80 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f0a6e3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04112a10 con 0x7f9f041083c0 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f08c80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f08c80640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44088/0 (socket says 172.21.15.16:44088) 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f08c80640 1 -- 172.21.15.16:0/2022738868 learned_addr learned my addr 172.21.15.16:0/2022738868 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f03fff640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f04102a80 0x7f9f0419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.093+0000 7f9f037fe640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f041083c0 0x7f9f0419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 -- 172.21.15.16:0/2022738868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f041083c0 msgr2=0x7f9f0419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f041083c0 0x7f9f0419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 -- 172.21.15.16:0/2022738868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f04102a80 msgr2=0x7f9f0419a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f04102a80 0x7f9f0419a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 -- 172.21.15.16:0/2022738868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f041a0280 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f08c80640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00b970 tx=0x7f9efc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc019070 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9efc014440 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f041a0570 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc0075d0 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.094+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f041a6d90 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.095+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f04103d00 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.097+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f9efc01c070 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.097+0000 7f9f017fa640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 0x7f9eec0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.098+0000 7f9f03fff640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 0x7f9eec0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.098+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7712+0+0 (secure 0 0 0) 0x7f9efc0e6930 con 0x7f9f04108d10 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.098+0000 7f9f03fff640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 0x7f9eec0b8aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40057c0 tx=0x7f9ef40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.101+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9efc0b1800 con 0x7f9f04108d10 2024-09-21T02:04:32.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.289+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f9f04102e90 con 0x7f9f04108d10 2024-09-21T02:04:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.606+0000 7f9f017fa640 1 -- 172.21.15.16:0/2022738868 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v74) ==== 171+0+0 (secure 0 0 0) 0x7f9efc0b6740 con 0x7f9f04108d10 2024-09-21T02:04:32.607 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-09-21T02:04:32.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 msgr2=0x7f9eec0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 0x7f9eec0b8aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40057c0 tx=0x7f9ef40058d0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 msgr2=0x7f9f0419fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00b970 tx=0x7f9efc00bd70 comp rx=0 tx=0).stop 2024-09-21T02:04:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 shutdown_connections 2024-09-21T02:04:32.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9eec0b6590 0x7f9eec0b8aa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f04108d10 0x7f9f0419fc10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f041083c0 0x7f9f0419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 --2- 172.21.15.16:0/2022738868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f04102a80 0x7f9f0419a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 >> 172.21.15.16:0/2022738868 conn(0x7f9f04079740 msgr2=0x7f9f04100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 shutdown_connections 2024-09-21T02:04:32.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:32.609+0000 7f9f0a6e3640 1 -- 172.21.15.16:0/2022738868 wait complete. 2024-09-21T02:04:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:32 smithi016 ceph-mon[31245]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:04:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:32 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:04:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:32 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:04:32.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:32 smithi016 ceph-mon[31245]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:04:32.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:32 smithi161 ceph-mon[34061]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:04:32.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:32 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:04:32.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:32 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:04:32.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:32 smithi161 ceph-mon[34061]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:04:33.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:32 smithi123 ceph-mon[33453]: osdmap e73: 12 total, 12 up, 12 in 2024-09-21T02:04:33.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:32 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} : dispatch 2024-09-21T02:04:33.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:32 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2022738868' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-21T02:04:33.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:32 smithi123 ceph-mon[33453]: osdmap e74: 12 total, 12 up, 12 in 2024-09-21T02:04:33.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-21T02:04:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:33.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:33 smithi016 ceph-mon[31245]: pgmap v261: 33 pgs: 1 creating+activating, 15 creating+peering, 1 active+clean, 16 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:33.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:33 smithi016 ceph-mon[31245]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:04:33.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:33 smithi161 ceph-mon[34061]: pgmap v261: 33 pgs: 1 creating+activating, 15 creating+peering, 1 active+clean, 16 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:33.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:33 smithi161 ceph-mon[34061]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:04:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:33 smithi123 ceph-mon[33453]: pgmap v261: 33 pgs: 1 creating+activating, 15 creating+peering, 1 active+clean, 16 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:34.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:33 smithi123 ceph-mon[33453]: osdmap e75: 12 total, 12 up, 12 in 2024-09-21T02:04:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- 172.21.15.16:0/3834744369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c141094a0 msgr2=0x7f3c141098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 --2- 172.21.15.16:0/3834744369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c141094a0 0x7f3c141098b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009900 tx=0x7f3c0802a410 comp rx=0 tx=0).stop 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- 172.21.15.16:0/3834744369 shutdown_connections 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 --2- 172.21.15.16:0/3834744369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c14109df0 0x7f3c14110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 --2- 172.21.15.16:0/3834744369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c141094a0 0x7f3c141098b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 --2- 172.21.15.16:0/3834744369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14103b60 0x7f3c14103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- 172.21.15.16:0/3834744369 >> 172.21.15.16:0/3834744369 conn(0x7f3c1407b850 msgr2=0x7f3c1407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- 172.21.15.16:0/3834744369 shutdown_connections 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- 172.21.15.16:0/3834744369 wait complete. 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 Processor -- start 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.121+0000 7f3c1a363640 1 -- start start 2024-09-21T02:04:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 0x7f3c1419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14109df0 0x7f3c1419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14113960 con 0x7f3c14103b60 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141137e0 con 0x7f3c14109df0 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c1a363640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14113ae0 con 0x7f3c141094a0 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44102/0 (socket says 172.21.15.16:44102) 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 -- 172.21.15.16:0/2972437848 learned_addr learned my addr 172.21.15.16:0/2972437848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c137fe640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 0x7f3c1419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c18900640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14109df0 0x7f3c1419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 -- 172.21.15.16:0/2972437848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 msgr2=0x7f3c1419a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 0x7f3c1419a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 -- 172.21.15.16:0/2972437848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14109df0 msgr2=0x7f3c1419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14109df0 0x7f3c1419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 -- 172.21.15.16:0/2972437848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c141a0380 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c137fe640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 0x7f3c1419a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.122+0000 7f3c13fff640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400c890 tx=0x7f3c0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.123+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04004220 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.123+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c141a0670 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.123+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c04007e40 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.123+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04004a00 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.123+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c141a6dd0 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.124+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c14103f70 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.125+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f3c040079b0 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.126+0000 7f3c117fa640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 0x7f3bfc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.126+0000 7f3c137fe640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 0x7f3bfc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.126+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 8094+0+0 (secure 0 0 0) 0x7f3c040e7460 con 0x7f3c14103b60 2024-09-21T02:04:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.127+0000 7f3c137fe640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 0x7f3bfc0b8a70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3c080057c0 tx=0x7f3c08009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.130+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c04014030 con 0x7f3c14103b60 2024-09-21T02:04:34.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.357+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3c1419b540 con 0x7f3c14103b60 2024-09-21T02:04:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.646+0000 7f3c117fa640 1 -- 172.21.15.16:0/2972437848 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3c1419b540 con 0x7f3c14103b60 2024-09-21T02:04:34.648 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-09-21T02:04:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 msgr2=0x7f3bfc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 0x7f3bfc0b8a70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3c080057c0 tx=0x7f3c08009680 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 msgr2=0x7f3c1419a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400c890 tx=0x7f3c0400cd50 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 shutdown_connections 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bfc0b6560 0x7f3bfc0b8a70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c14109df0 0x7f3c1419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c141094a0 0x7f3c1419a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 --2- 172.21.15.16:0/2972437848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c14103b60 0x7f3c1419a410 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 >> 172.21.15.16:0/2972437848 conn(0x7f3c1407b850 msgr2=0x7f3c141075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 shutdown_connections 2024-09-21T02:04:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:34.648+0000 7f3c1a363640 1 -- 172.21.15.16:0/2972437848 wait complete. 2024-09-21T02:04:34.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:34 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:04:34.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:34 smithi016 ceph-mon[31245]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T02:04:34.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:34 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:04:34.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:34 smithi161 ceph-mon[34061]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T02:04:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:34 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd={"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} : dispatch 2024-09-21T02:04:35.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:34 smithi123 ceph-mon[33453]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-21T02:04:35.378 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-21T02:04:35.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-21T02:04:35.646 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:35 smithi016 ceph-mon[31245]: pgmap v264: 97 pgs: 1 creating+activating, 19 creating+peering, 31 active+clean, 46 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:35 smithi016 ceph-mon[31245]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:04:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:35 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:04:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:35 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T02:04:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:35 smithi016 ceph-mon[31245]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T02:04:35.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:35 smithi161 ceph-mon[34061]: pgmap v264: 97 pgs: 1 creating+activating, 19 creating+peering, 31 active+clean, 46 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:35.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:35 smithi161 ceph-mon[34061]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:04:35.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:35 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:04:35.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:35 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T02:04:35.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:35 smithi161 ceph-mon[34061]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T02:04:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:35 smithi123 ceph-mon[33453]: pgmap v264: 97 pgs: 1 creating+activating, 19 creating+peering, 31 active+clean, 46 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:35 smithi123 ceph-mon[33453]: osdmap e76: 12 total, 12 up, 12 in 2024-09-21T02:04:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:35 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2972437848' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-21T02:04:36.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:35 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-21T02:04:36.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:35 smithi123 ceph-mon[33453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-21T02:04:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.192+0000 7f66cadb2640 1 -- 172.21.15.16:0/1730529724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c4102a60 msgr2=0x7f66c4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.192+0000 7f66cadb2640 1 --2- 172.21.15.16:0/1730529724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c4102a60 0x7f66c4102e70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009900 tx=0x7f66c002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 -- 172.21.15.16:0/1730529724 shutdown_connections 2024-09-21T02:04:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 --2- 172.21.15.16:0/1730529724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4108cf0 0x7f66c410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 --2- 172.21.15.16:0/1730529724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c41083a0 0x7f66c41087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 --2- 172.21.15.16:0/1730529724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c4102a60 0x7f66c4102e70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 -- 172.21.15.16:0/1730529724 >> 172.21.15.16:0/1730529724 conn(0x7f66c4079730 msgr2=0x7f66c4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 -- 172.21.15.16:0/1730529724 shutdown_connections 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 -- 172.21.15.16:0/1730529724 wait complete. 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 Processor -- start 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.193+0000 7f66cadb2640 1 -- start start 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 0x7f66c419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4108cf0 0x7f66c419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112810 con 0x7f66c41083a0 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112690 con 0x7f66c4102a60 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4112990 con 0x7f66c4108cf0 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44110/0 (socket says 172.21.15.16:44110) 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 -- 172.21.15.16:0/2842441820 learned_addr learned my addr 172.21.15.16:0/2842441820 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66c8b4e640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 0x7f66c419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66c934f640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4108cf0 0x7f66c419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 -- 172.21.15.16:0/2842441820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4108cf0 msgr2=0x7f66c419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4108cf0 0x7f66c419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 -- 172.21.15.16:0/2842441820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 msgr2=0x7f66c419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 0x7f66c419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 -- 172.21.15.16:0/2842441820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c41a0260 con 0x7f66c41083a0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66c8b4e640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 0x7f66c419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66bbfff640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f66b400d740 tx=0x7f66b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4017070 con 0x7f66c41083a0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c41a0550 con 0x7f66c41083a0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66b400de50 con 0x7f66c41083a0 2024-09-21T02:04:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.194+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4004c60 con 0x7f66c41083a0 2024-09-21T02:04:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.195+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c41a6d70 con 0x7f66c41083a0 2024-09-21T02:04:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.196+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c4103ce0 con 0x7f66c41083a0 2024-09-21T02:04:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.197+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f66b4004e00 con 0x7f66c41083a0 2024-09-21T02:04:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.198+0000 7f66b9ffb640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 0x7f66ac0b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.198+0000 7f66c8b4e640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 0x7f66ac0b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.199+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8194+0+0 (secure 0 0 0) 0x7f66b40e71a0 con 0x7f66c41083a0 2024-09-21T02:04:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.199+0000 7f66c8b4e640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 0x7f66ac0b8aa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006010 tx=0x7f66c0000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:36.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.202+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f66b40ada50 con 0x7f66c41083a0 2024-09-21T02:04:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.400+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f66c41a6f40 con 0x7f66c41083a0 2024-09-21T02:04:36.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:36 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:active 2024-09-21T02:04:36.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:36 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:36.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:36 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:36.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:36 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:04:36.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.661+0000 7f66b9ffb640 1 -- 172.21.15.16:0/2842441820 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v77) ==== 157+0+0 (secure 0 0 0) 0x7f66b40b6020 con 0x7f66c41083a0 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 msgr2=0x7f66ac0b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 0x7f66ac0b8aa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006010 tx=0x7f66c0000bc0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 msgr2=0x7f66c419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f66b400d740 tx=0x7f66b400dc00 comp rx=0 tx=0).stop 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 shutdown_connections 2024-09-21T02:04:36.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66ac0b6590 0x7f66ac0b8aa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4108cf0 0x7f66c419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66c41083a0 0x7f66c419a950 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 --2- 172.21.15.16:0/2842441820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c4102a60 0x7f66c419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 >> 172.21.15.16:0/2842441820 conn(0x7f66c4079730 msgr2=0x7f66c4101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 shutdown_connections 2024-09-21T02:04:36.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:36.664+0000 7f66cadb2640 1 -- 172.21.15.16:0/2842441820 wait complete. 2024-09-21T02:04:36.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:36 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:active 2024-09-21T02:04:36.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:36 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:36.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:36 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:36.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:36 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:04:37.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:36 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] up:active 2024-09-21T02:04:37.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:36 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:37.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:36 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:37.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:36 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} : dispatch 2024-09-21T02:04:37.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-21T02:04:37.651 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:37 smithi123 ceph-mon[33453]: pgmap v266: 97 pgs: 1 creating+activating, 7 creating+peering, 69 active+clean, 20 unknown; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 232 B/s wr, 5 op/s 2024-09-21T02:04:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:37 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:04:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:37 smithi123 ceph-mon[33453]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:04:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:37 smithi161 ceph-mon[34061]: pgmap v266: 97 pgs: 1 creating+activating, 7 creating+peering, 69 active+clean, 20 unknown; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 232 B/s wr, 5 op/s 2024-09-21T02:04:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:37 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:04:37.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:37 smithi161 ceph-mon[34061]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:04:37.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:37 smithi016 ceph-mon[31245]: pgmap v266: 97 pgs: 1 creating+activating, 7 creating+peering, 69 active+clean, 20 unknown; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 232 B/s wr, 5 op/s 2024-09-21T02:04:37.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:37 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2842441820' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-21T02:04:37.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:37 smithi016 ceph-mon[31245]: osdmap e77: 12 total, 12 up, 12 in 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 -- 172.21.15.16:0/1475821532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b70 msgr2=0x7f29a0103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 --2- 172.21.15.16:0/1475821532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b70 0x7f29a0103f80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2990009900 tx=0x7f299002a410 comp rx=0 tx=0).stop 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 -- 172.21.15.16:0/1475821532 shutdown_connections 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 --2- 172.21.15.16:0/1475821532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a0110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 --2- 172.21.15.16:0/1475821532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a01094b0 0x7f29a01098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:38.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 --2- 172.21.15.16:0/1475821532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b70 0x7f29a0103f80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:38.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 -- 172.21.15.16:0/1475821532 >> 172.21.15.16:0/1475821532 conn(0x7f29a00ff5a0 msgr2=0x7f29a01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:38.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 -- 172.21.15.16:0/1475821532 shutdown_connections 2024-09-21T02:04:38.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.148+0000 7f29a624f640 1 -- 172.21.15.16:0/1475821532 wait complete. 2024-09-21T02:04:38.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 Processor -- start 2024-09-21T02:04:38.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 -- start start 2024-09-21T02:04:38.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:38.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a01094b0 0x7f29a019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0113970 con 0x7f29a0103b70 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29a01137f0 con 0x7f29a0109e00 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f29a624f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0113af0 con 0x7f29a01094b0 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f299f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f299f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48446/0 (socket says 172.21.15.16:48446) 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f299f7fe640 1 -- 172.21.15.16:0/1836415727 learned_addr learned my addr 172.21.15.16:0/1836415727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f299effd640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a01094b0 0x7f29a019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.149+0000 7f299ffff640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 -- 172.21.15.16:0/1836415727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a01094b0 msgr2=0x7f29a019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a01094b0 0x7f29a019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 -- 172.21.15.16:0/1836415727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 msgr2=0x7f29a019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 -- 172.21.15.16:0/1836415727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a01a02f0 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299ffff640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299f7fe640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2990009230 tx=0x7f29900319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299003f030 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a01a0580 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29900043d0 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990035070 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.150+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a00755f0 con 0x7f29a0103b70 2024-09-21T02:04:38.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.151+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a0104df0 con 0x7f29a0103b70 2024-09-21T02:04:38.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.153+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f2990042030 con 0x7f29a0103b70 2024-09-21T02:04:38.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.154+0000 7f299cff9640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 0x7f29880b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:38.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.154+0000 7f299effd640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 0x7f29880b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:38.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.154+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8492+0+0 (secure 0 0 0) 0x7f2990107c40 con 0x7f29a0103b70 2024-09-21T02:04:38.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.154+0000 7f299effd640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 0x7f29880b8a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f29a0108f60 tx=0x7f2994005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.157+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f29900d27b0 con 0x7f29a0103b70 2024-09-21T02:04:38.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:38.346+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f29a010bbd0 con 0x7f29a0103b70 2024-09-21T02:04:38.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:38 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:04:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:38 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:04:39.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:38 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} : dispatch 2024-09-21T02:04:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.670+0000 7f299cff9640 1 -- 172.21.15.16:0/1836415727 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v79) ==== 228+0+0 (secure 0 0 0) 0x7f29900d76f0 con 0x7f29a0103b70 2024-09-21T02:04:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2024-09-21T02:04:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.673+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 msgr2=0x7f29880b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.673+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 0x7f29880b8a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f29a0108f60 tx=0x7f2994005f90 comp rx=0 tx=0).stop 2024-09-21T02:04:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 msgr2=0x7f29a019a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2990009230 tx=0x7f29900319a0 comp rx=0 tx=0).stop 2024-09-21T02:04:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 shutdown_connections 2024-09-21T02:04:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f29880b6560 0x7f29880b8a70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:39.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a0109e00 0x7f29a019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a01094b0 0x7f29a019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 --2- 172.21.15.16:0/1836415727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0103b70 0x7f29a019a380 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.674+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 >> 172.21.15.16:0/1836415727 conn(0x7f29a00ff5a0 msgr2=0x7f29a01019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.676+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 shutdown_connections 2024-09-21T02:04:39.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:39.676+0000 7f29a624f640 1 -- 172.21.15.16:0/1836415727 wait complete. 2024-09-21T02:04:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:39 smithi016 ceph-mon[31245]: pgmap v268: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T02:04:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:39 smithi016 ceph-mon[31245]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:04:40.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:39 smithi123 ceph-mon[33453]: pgmap v268: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T02:04:40.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:39 smithi123 ceph-mon[33453]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:04:40.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:39 smithi161 ceph-mon[34061]: pgmap v268: 97 pgs: 4 creating+peering, 93 active+clean; 577 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-21T02:04:40.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:39 smithi161 ceph-mon[34061]: osdmap e78: 12 total, 12 up, 12 in 2024-09-21T02:04:40.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-21T02:04:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:41.020 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:04:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:04:40.560+0000 7f4c33c91640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-21T02:04:41.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:40 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:04:41.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:40 smithi123 ceph-mon[33453]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:04:41.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:40 smithi123 ceph-mon[33453]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:04:41.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:40 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:04:41.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:40 smithi016 ceph-mon[31245]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:04:41.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:40 smithi016 ceph-mon[31245]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:04:41.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:40 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1836415727' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-21T02:04:41.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:40 smithi161 ceph-mon[34061]: osdmap e79: 12 total, 12 up, 12 in 2024-09-21T02:04:41.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:40 smithi161 ceph-mon[34061]: osdmap e80: 12 total, 12 up, 12 in 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 -- 172.21.15.16:0/34583229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 msgr2=0x7f6d5c102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 --2- 172.21.15.16:0/34583229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c102640 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009900 tx=0x7f6d5802a410 comp rx=0 tx=0).stop 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 -- 172.21.15.16:0/34583229 shutdown_connections 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 --2- 172.21.15.16:0/34583229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 0x7f6d5c10ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 --2- 172.21.15.16:0/34583229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 --2- 172.21.15.16:0/34583229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c102640 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 -- 172.21.15.16:0/34583229 >> 172.21.15.16:0/34583229 conn(0x7f6d5c079e40 msgr2=0x7f6d5c07a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 -- 172.21.15.16:0/34583229 shutdown_connections 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 -- 172.21.15.16:0/34583229 wait complete. 2024-09-21T02:04:41.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.188+0000 7f6d63e70640 1 Processor -- start 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 -- start start 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 0x7f6d5c199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c1128b0 con 0x7f6d5c102230 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c112730 con 0x7f6d5c1084c0 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d63e70640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c112a30 con 0x7f6d5c107b70 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48462/0 (socket says 172.21.15.16:48462) 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 -- 172.21.15.16:0/2293985907 learned_addr learned my addr 172.21.15.16:0/2293985907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d6240d640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 0x7f6d5c199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d6140b640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 -- 172.21.15.16:0/2293985907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 msgr2=0x7f6d5c199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 -- 172.21.15.16:0/2293985907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 msgr2=0x7f6d5c199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 0x7f6d5c199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.189+0000 7f6d61c0c640 1 -- 172.21.15.16:0/2293985907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5c1a8690 con 0x7f6d5c102230 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d6140b640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d61c0c640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009230 tx=0x7f6d58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5803f030 con 0x7f6d5c102230 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d58003a80 con 0x7f6d5c102230 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5c1a88c0 con 0x7f6d5c102230 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58035070 con 0x7f6d5c102230 2024-09-21T02:04:41.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.190+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5c1a8da0 con 0x7f6d5c102230 2024-09-21T02:04:41.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.191+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5c102640 con 0x7f6d5c102230 2024-09-21T02:04:41.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.193+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f6d58042030 con 0x7f6d5c102230 2024-09-21T02:04:41.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.194+0000 7f6d52ffd640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 0x7f6d480b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:41.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.194+0000 7f6d6140b640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 0x7f6d480b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:41.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.194+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8985+0+0 (secure 0 0 0) 0x7f6d58108240 con 0x7f6d5c102230 2024-09-21T02:04:41.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.194+0000 7f6d6140b640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 0x7f6d480b8a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0058c0 tx=0x7f6d4c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:41.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.197+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6d580ce780 con 0x7f6d5c102230 2024-09-21T02:04:41.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.383+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f6d5c10fac0 con 0x7f6d5c102230 2024-09-21T02:04:41.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.690+0000 7f6d52ffd640 1 -- 172.21.15.16:0/2293985907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v81) ==== 169+0+0 (secure 0 0 0) 0x7f6d5c10fac0 con 0x7f6d5c102230 2024-09-21T02:04:41.691 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 4 allow_ec_overwrites to true 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 msgr2=0x7f6d480b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 0x7f6d480b8a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0058c0 tx=0x7f6d4c005850 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 msgr2=0x7f6d5c199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009230 tx=0x7f6d58004270 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 shutdown_connections 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6d480b6560 0x7f6d480b8a70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d5c1084c0 0x7f6d5c199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d5c107b70 0x7f6d5c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 --2- 172.21.15.16:0/2293985907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c102230 0x7f6d5c199010 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 >> 172.21.15.16:0/2293985907 conn(0x7f6d5c079e40 msgr2=0x7f6d5c105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:41.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 shutdown_connections 2024-09-21T02:04:41.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:41.693+0000 7f6d63e70640 1 -- 172.21.15.16:0/2293985907 wait complete. 2024-09-21T02:04:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:41 smithi123 ceph-mon[33453]: pgmap v271: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-21T02:04:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:41 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:04:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:41 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:04:42.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:41 smithi123 ceph-mon[33453]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:04:42.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:41 smithi016 ceph-mon[31245]: pgmap v271: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-21T02:04:42.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:41 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:04:42.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:41 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:04:42.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:41 smithi016 ceph-mon[31245]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:04:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:41 smithi161 ceph-mon[34061]: pgmap v271: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-21T02:04:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:41 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-21T02:04:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:41 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2293985907' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-21T02:04:42.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:41 smithi161 ceph-mon[34061]: osdmap e81: 12 total, 12 up, 12 in 2024-09-21T02:04:42.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-21T02:04:42.666 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 -- 172.21.15.16:0/1585247199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8102ae0 msgr2=0x7f53c8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1585247199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8102ae0 0x7f53c8102ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009900 tx=0x7f53c402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 -- 172.21.15.16:0/1585247199 shutdown_connections 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1585247199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c8108d20 0x7f53c810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1585247199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c81083d0 0x7f53c81087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1585247199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8102ae0 0x7f53c8102ef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.185+0000 7f53cf78a640 1 -- 172.21.15.16:0/1585247199 >> 172.21.15.16:0/1585247199 conn(0x7f53c8079760 msgr2=0x7f53c8079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.186+0000 7f53cf78a640 1 -- 172.21.15.16:0/1585247199 shutdown_connections 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.186+0000 7f53cf78a640 1 -- 172.21.15.16:0/1585247199 wait complete. 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.186+0000 7f53cf78a640 1 Processor -- start 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.186+0000 7f53cf78a640 1 -- start start 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c8102ae0 0x7f53c819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 0x7f53c819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8112840 con 0x7f53c8108d20 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81126c0 con 0x7f53c8102ae0 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cf78a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f53c81129c0 con 0x7f53c81083d0 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48486/0 (socket says 172.21.15.16:48486) 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 -- 172.21.15.16:0/1677490416 learned_addr learned my addr 172.21.15.16:0/1677490416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cd526640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c8102ae0 0x7f53c819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53ccd25640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 0x7f53c819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 -- 172.21.15.16:0/1677490416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 msgr2=0x7f53c819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 0x7f53c819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 -- 172.21.15.16:0/1677490416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c8102ae0 msgr2=0x7f53c819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c8102ae0 0x7f53c819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 -- 172.21.15.16:0/1677490416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c81a02b0 con 0x7f53c8108d20 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53ccd25640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 0x7f53c819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53cdd27640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eaa0 tx=0x7f53c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c000cdb0 con 0x7f53c8108d20 2024-09-21T02:04:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.187+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53c0005020 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.188+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c81a05a0 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.188+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c00116d0 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.188+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c81a6d00 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.189+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c8103d10 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.190+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f53c00051d0 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.191+0000 7f53be7fc640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 0x7f53b40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.191+0000 7f53cd526640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 0x7f53b40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.191+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8985+0+0 (secure 0 0 0) 0x7f53c00e7b50 con 0x7f53c8108d20 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.192+0000 7f53cd526640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 0x7f53b40b8af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53c40057c0 tx=0x7f53c40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.194+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f53c0016030 con 0x7f53c8108d20 2024-09-21T02:04:43.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.423+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f53c819b470 con 0x7f53c8108d20 2024-09-21T02:04:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.702+0000 7f53be7fc640 1 -- 172.21.15.16:0/1677490416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f53c00ae170 con 0x7f53c8108d20 2024-09-21T02:04:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 4 to fsmap 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.705+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 msgr2=0x7f53b40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.705+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 0x7f53b40b8af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53c40057c0 tx=0x7f53c40058d0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.705+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 msgr2=0x7f53c819fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.705+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eaa0 tx=0x7f53c000ef60 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 shutdown_connections 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f53b40b65e0 0x7f53b40b8af0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53c8108d20 0x7f53c819fb80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53c81083d0 0x7f53c819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 --2- 172.21.15.16:0/1677490416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53c8102ae0 0x7f53c819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 >> 172.21.15.16:0/1677490416 conn(0x7f53c8079760 msgr2=0x7f53c8101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 shutdown_connections 2024-09-21T02:04:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:43.706+0000 7f53cf78a640 1 -- 172.21.15.16:0/1677490416 wait complete. 2024-09-21T02:04:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:43 smithi123 ceph-mon[33453]: pgmap v274: 129 pgs: 2 creating+activating, 20 unknown, 107 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T02:04:44.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:43 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:04:44.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:43 smithi016 ceph-mon[31245]: pgmap v274: 129 pgs: 2 creating+activating, 20 unknown, 107 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T02:04:44.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:43 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:04:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:43 smithi161 ceph-mon[34061]: pgmap v274: 129 pgs: 2 creating+activating, 20 unknown, 107 active+clean; 579 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-21T02:04:44.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:43 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd={"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} : dispatch 2024-09-21T02:04:44.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:44.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:44 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:04:44.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:44 smithi016 ceph-mon[31245]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:04:44.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:44 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:44 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:04:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:44 smithi123 ceph-mon[33453]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:04:45.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:44 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:45.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:44 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1677490416' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-21T02:04:45.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:44 smithi161 ceph-mon[34061]: osdmap e82: 12 total, 12 up, 12 in 2024-09-21T02:04:45.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:44 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:04:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.208+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3877071280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0109df0 msgr2=0x7fe3e0110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.208+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3877071280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0109df0 0x7fe3e0110620 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009ca0 tx=0x7fe3d802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3877071280 shutdown_connections 2024-09-21T02:04:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3877071280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0109df0 0x7fe3e0110620 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3877071280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e01094a0 0x7fe3e01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3877071280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e0103b60 0x7fe3e0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3877071280 >> 172.21.15.16:0/3877071280 conn(0x7fe3e007b850 msgr2=0x7fe3e007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3877071280 shutdown_connections 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3877071280 wait complete. 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 Processor -- start 2024-09-21T02:04:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.209+0000 7fe3e6fd6640 1 -- start start 2024-09-21T02:04:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 0x7fe3e019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e0109df0 0x7fe3e019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01139b0 con 0x7fe3e0103b60 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0113830 con 0x7fe3e01094a0 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e6fd6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0113b30 con 0x7fe3e0109df0 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 0x7fe3e019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48496/0 (socket says 172.21.15.16:48496) 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 0x7fe3e019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:44226/0 (socket says 172.21.15.16:44226) 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 -- 172.21.15.16:0/3215558477 learned_addr learned my addr 172.21.15.16:0/3215558477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e5573640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e0109df0 0x7fe3e019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 -- 172.21.15.16:0/3215558477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e0109df0 msgr2=0x7fe3e019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e0109df0 0x7fe3e019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 -- 172.21.15.16:0/3215558477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 msgr2=0x7fe3e019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 0x7fe3e019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.210+0000 7fe3e4d72640 1 -- 172.21.15.16:0/3215558477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e01a0370 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3e4d72640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c920 tx=0x7fe3d400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4007c70 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe3d4007e10 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4018710 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e01a0660 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.211+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e01a6e80 con 0x7fe3e0103b60 2024-09-21T02:04:45.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.212+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e0103f70 con 0x7fe3e0103b60 2024-09-21T02:04:45.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.214+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fe3d400f450 con 0x7fe3e0103b60 2024-09-21T02:04:45.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.214+0000 7fe3ddffb640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 0x7fe3d00b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:45.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.215+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9017+0+0 (secure 0 0 0) 0x7fe3d40e7980 con 0x7fe3e0103b60 2024-09-21T02:04:45.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.215+0000 7fe3dffff640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 0x7fe3d00b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:45.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.215+0000 7fe3dffff640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 0x7fe3d00b8b20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0109220 tx=0x7fe3cc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.218+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe3d4014030 con 0x7fe3e0103b60 2024-09-21T02:04:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.446+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3e019b5f0 con 0x7fe3e0103b60 2024-09-21T02:04:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.446+0000 7fe3ddffb640 1 -- 172.21.15.16:0/3215558477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11857 (secure 0 0 0) 0x7fe3d40ae010 con 0x7fe3e0103b60 2024-09-21T02:04:45.447 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:45.447 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"btime":"2024-09-21T02:04:43:701558+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14841,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14853,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24667,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24683,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:04:43.425228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24710,"qdb_cluster":[24710]},"id":1}]} 2024-09-21T02:04:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 17 2024-09-21T02:04:45.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 msgr2=0x7fe3d00b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 0x7fe3d00b8b20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0109220 tx=0x7fe3cc00b390 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 msgr2=0x7fe3e019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c920 tx=0x7fe3d400cde0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 shutdown_connections 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe3d00b6610 0x7fe3d00b8b20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3e0109df0 0x7fe3e019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e01094a0 0x7fe3e019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 --2- 172.21.15.16:0/3215558477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e0103b60 0x7fe3e019a4c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.448+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 >> 172.21.15.16:0/3215558477 conn(0x7fe3e007b850 msgr2=0x7fe3e0107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.449+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 shutdown_connections 2024-09-21T02:04:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:45.449+0000 7fe3e6fd6640 1 -- 172.21.15.16:0/3215558477 wait complete. 2024-09-21T02:04:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:45 smithi161 ceph-mon[34061]: pgmap v276: 129 pgs: 2 creating+activating, 127 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:45 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3215558477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:46.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:45 smithi123 ceph-mon[33453]: pgmap v276: 129 pgs: 2 creating+activating, 127 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:46.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:45 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3215558477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:45 smithi016 ceph-mon[31245]: pgmap v276: 129 pgs: 2 creating+activating, 127 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:45 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3215558477' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:46.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:04:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- 172.21.15.16:0/3165174459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 msgr2=0x7f7e00110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 --2- 172.21.15.16:0/3165174459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e00110620 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009ca0 tx=0x7f7df802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- 172.21.15.16:0/3165174459 shutdown_connections 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 --2- 172.21.15.16:0/3165174459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e00110620 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 --2- 172.21.15.16:0/3165174459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e001094a0 0x7f7e001098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 --2- 172.21.15.16:0/3165174459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e00103b60 0x7f7e00103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- 172.21.15.16:0/3165174459 >> 172.21.15.16:0/3165174459 conn(0x7f7e0007b850 msgr2=0x7f7e0007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- 172.21.15.16:0/3165174459 shutdown_connections 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- 172.21.15.16:0/3165174459 wait complete. 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 Processor -- start 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.060+0000 7f7e08432640 1 -- start start 2024-09-21T02:04:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e00103b60 0x7f7e0019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e001094a0 0x7f7e0019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001139b0 con 0x7f7e00109df0 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00113830 con 0x7f7e001094a0 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e08432640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00113b30 con 0x7f7e00103b60 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48518/0 (socket says 172.21.15.16:48518) 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 -- 172.21.15.16:0/778576758 learned_addr learned my addr 172.21.15.16:0/778576758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e059cd640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e001094a0 0x7f7e0019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e061ce640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e00103b60 0x7f7e0019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 -- 172.21.15.16:0/778576758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e00103b60 msgr2=0x7f7e0019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e00103b60 0x7f7e0019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 -- 172.21.15.16:0/778576758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e001094a0 msgr2=0x7f7e0019a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e001094a0 0x7f7e0019a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 -- 172.21.15.16:0/778576758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e001a0310 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.061+0000 7f7e069cf640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005cc0 tx=0x7f7df8032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.062+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8032e00 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.062+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7df80044d0 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.062+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e001a05a0 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.062+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8030b70 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.062+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e001a6d60 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.063+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e00103f70 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.065+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f7df8030d10 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.065+0000 7f7df77fe640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 0x7f7dec0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.065+0000 7f7e061ce640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 0x7f7dec0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.065+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9017+0+0 (secure 0 0 0) 0x7f7df8108ac0 con 0x7f7e00109df0 2024-09-21T02:04:47.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.066+0000 7f7e061ce640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 0x7f7dec0b8a70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc005fd0 tx=0x7f7dfc005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:47.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.069+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7df80cf0d0 con 0x7f7e00109df0 2024-09-21T02:04:47.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.258+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e0019b4d0 con 0x7f7e00109df0 2024-09-21T02:04:47.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.259+0000 7f7df77fe640 1 -- 172.21.15.16:0/778576758 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24530 (secure 0 0 0) 0x7f7e0019b4d0 con 0x7f7e00109df0 2024-09-21T02:04:47.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:47.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:43.425075+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 msgr2=0x7f7dec0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 0x7f7dec0b8a70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc005fd0 tx=0x7f7dfc005850 comp rx=0 tx=0).stop 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 msgr2=0x7f7e0019fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005cc0 tx=0x7f7df8032a10 comp rx=0 tx=0).stop 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 shutdown_connections 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7dec0b6560 0x7f7dec0b8a70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e00109df0 0x7f7e0019fbe0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e001094a0 0x7f7e0019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 --2- 172.21.15.16:0/778576758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e00103b60 0x7f7e0019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 >> 172.21.15.16:0/778576758 conn(0x7f7e0007b850 msgr2=0x7f7e001054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 shutdown_connections 2024-09-21T02:04:47.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:47.262+0000 7f7e08432640 1 -- 172.21.15.16:0/778576758 wait complete. 2024-09-21T02:04:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:47 smithi016 ceph-mon[31245]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:47 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/778576758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:48.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:47 smithi123 ceph-mon[33453]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:48.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:47 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/778576758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:48.177 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:47 smithi161 ceph-mon[34061]: pgmap v277: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:48.177 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:47 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/778576758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:48.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:04:48.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.927+0000 7f7043e8a640 1 -- 172.21.15.16:0/213808003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c103b60 msgr2=0x7f703c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.927+0000 7f7043e8a640 1 --2- 172.21.15.16:0/213808003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c103b60 0x7f703c103f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7038009ca0 tx=0x7f703802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 -- 172.21.15.16:0/213808003 shutdown_connections 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 --2- 172.21.15.16:0/213808003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 0x7f703c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 --2- 172.21.15.16:0/213808003 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c1094a0 0x7f703c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 --2- 172.21.15.16:0/213808003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c103b60 0x7f703c103f70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 -- 172.21.15.16:0/213808003 >> 172.21.15.16:0/213808003 conn(0x7f703c07b850 msgr2=0x7f703c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 -- 172.21.15.16:0/213808003 shutdown_connections 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.928+0000 7f7043e8a640 1 -- 172.21.15.16:0/213808003 wait complete. 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 Processor -- start 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 -- start start 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 0x7f703c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 0x7f703c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1139b0 con 0x7f703c1094a0 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f703c113830 con 0x7f703c109df0 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7043e8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f703c113b30 con 0x7f703c103b60 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40478/0 (socket says 172.21.15.16:40478) 2024-09-21T02:04:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 -- 172.21.15.16:0/2516469547 learned_addr learned my addr 172.21.15.16:0/2516469547 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7042427640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 0x7f703c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041c26640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 0x7f703c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 -- 172.21.15.16:0/2516469547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 msgr2=0x7f703c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 0x7f703c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 -- 172.21.15.16:0/2516469547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 msgr2=0x7f703c19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 0x7f703c19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041425640 1 -- 172.21.15.16:0/2516469547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c1a03d0 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.929+0000 7f7041c26640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 0x7f703c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7041425640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f702c00e9f0 tx=0x7f702c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c00ccc0 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f702c0108c0 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c018750 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c1a06c0 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.930+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c1a6e20 con 0x7f703c1094a0 2024-09-21T02:04:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.931+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703c103f70 con 0x7f703c1094a0 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.933+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f702c010430 con 0x7f703c1094a0 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.934+0000 7f7032ffd640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 0x7f70280b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.934+0000 7f7041c26640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 0x7f70280b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.934+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9017+0+0 (secure 0 0 0) 0x7f702c0e7fa0 con 0x7f703c1094a0 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.935+0000 7f7041c26640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 0x7f70280b8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7038005e30 tx=0x7f7038005fd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:48.937+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f702c0ae4c0 con 0x7f703c1094a0 2024-09-21T02:04:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.126+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f703c19b590 con 0x7f703c1094a0 2024-09-21T02:04:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.128+0000 7f7032ffd640 1 -- 172.21.15.16:0/2516469547 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24530 (secure 0 0 0) 0x7f703c19b590 con 0x7f703c1094a0 2024-09-21T02:04:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:49.129 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:43.425075+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:04:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 msgr2=0x7f70280b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 0x7f70280b8af0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7038005e30 tx=0x7f7038005fd0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 msgr2=0x7f703c19aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f702c00e9f0 tx=0x7f702c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 shutdown_connections 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f70280b65e0 0x7f70280b8af0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f703c109df0 0x7f703c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f703c1094a0 0x7f703c19aa00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 --2- 172.21.15.16:0/2516469547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f703c103b60 0x7f703c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.130+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 >> 172.21.15.16:0/2516469547 conn(0x7f703c07b850 msgr2=0x7f703c107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.131+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 shutdown_connections 2024-09-21T02:04:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:49.131+0000 7f7043e8a640 1 -- 172.21.15.16:0/2516469547 wait complete. 2024-09-21T02:04:49.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:49 smithi016 ceph-mon[31245]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:49.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:49 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2516469547' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:49.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:04:50.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:49 smithi123 ceph-mon[33453]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:50.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:49 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2516469547' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:50.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:49 smithi161 ceph-mon[34061]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:50.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:49 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2516469547' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:50.173 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 -- 172.21.15.16:0/4185283651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108420 msgr2=0x7f4170108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 --2- 172.21.15.16:0/4185283651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108420 0x7f4170108830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4164009900 tx=0x7f416402a410 comp rx=0 tx=0).stop 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 -- 172.21.15.16:0/4185283651 shutdown_connections 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 --2- 172.21.15.16:0/4185283651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170108d70 0x7f417010f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 --2- 172.21.15.16:0/4185283651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108420 0x7f4170108830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 --2- 172.21.15.16:0/4185283651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170102ae0 0x7f4170102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 -- 172.21.15.16:0/4185283651 >> 172.21.15.16:0/4185283651 conn(0x7f4170079770 msgr2=0x7f4170079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 -- 172.21.15.16:0/4185283651 shutdown_connections 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.662+0000 7f41767f3640 1 -- 172.21.15.16:0/4185283651 wait complete. 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 Processor -- start 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 -- start start 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170102ae0 0x7f417019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170108420 0x7f417019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4170113a00 con 0x7f4170108d70 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4170113880 con 0x7f4170102ae0 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.663+0000 7f41767f3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4170113b80 con 0x7f4170108420 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40496/0 (socket says 172.21.15.16:40496) 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 -- 172.21.15.16:0/1999920765 learned_addr learned my addr 172.21.15.16:0/1999920765 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f416f7fe640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170108420 0x7f417019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f416ffff640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170102ae0 0x7f417019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 -- 172.21.15.16:0/1999920765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170108420 msgr2=0x7f417019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170108420 0x7f417019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 -- 172.21.15.16:0/1999920765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170102ae0 msgr2=0x7f417019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170102ae0 0x7f417019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 -- 172.21.15.16:0/1999920765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41701a03a0 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f4174d90640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f416800eaa0 tx=0x7f416800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416800cdb0 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41701a0690 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4168005020 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41680116d0 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.664+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41701a6df0 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.665+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4170103d60 con 0x7f4170108d70 2024-09-21T02:04:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.668+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f41680051d0 con 0x7f4170108d70 2024-09-21T02:04:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.669+0000 7f416d7fa640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 0x7f41580b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.669+0000 7f416ffff640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 0x7f41580b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.669+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9017+0+0 (secure 0 0 0) 0x7f41680e7ce0 con 0x7f4170108d70 2024-09-21T02:04:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.669+0000 7f416ffff640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 0x7f41580b8a70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f41600058c0 tx=0x7f416000a4c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:50.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.672+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4168016030 con 0x7f4170108d70 2024-09-21T02:04:50.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.902+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4170102ef0 con 0x7f4170108d70 2024-09-21T02:04:50.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.902+0000 7f416d7fa640 1 -- 172.21.15.16:0/1999920765 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11857 (secure 0 0 0) 0x7f4170102ef0 con 0x7f4170108d70 2024-09-21T02:04:50.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:50.904 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"btime":"2024-09-21T02:04:43:701558+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14841,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14853,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24595,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24626,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24643,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24655,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24659,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24667,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24683,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:04:43.425228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24710,"qdb_cluster":[24710]},"id":1}]} 2024-09-21T02:04:50.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 17 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 msgr2=0x7f41580b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 0x7f41580b8a70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f41600058c0 tx=0x7f416000a4c0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 msgr2=0x7f417019fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f416800eaa0 tx=0x7f416800ef60 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 shutdown_connections 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f41580b6560 0x7f41580b8a70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4170108d70 0x7f417019fc70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4170108420 0x7f417019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 --2- 172.21.15.16:0/1999920765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4170102ae0 0x7f417019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 >> 172.21.15.16:0/1999920765 conn(0x7f4170079770 msgr2=0x7f4170106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 shutdown_connections 2024-09-21T02:04:50.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:50.905+0000 7f41767f3640 1 -- 172.21.15.16:0/1999920765 wait complete. 2024-09-21T02:04:51.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:04:51.970 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:51 smithi123 ceph-mon[33453]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:52.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:51 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1999920765' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:52.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:51 smithi016 ceph-mon[31245]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:52.091 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:51 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1999920765' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:52.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:51 smithi161 ceph-mon[34061]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:52.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:51 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1999920765' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:04:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 -- 172.21.15.16:0/2058435551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 msgr2=0x7f50cc10d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:52.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 --2- 172.21.15.16:0/2058435551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc10d8d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009900 tx=0x7f50c002a410 comp rx=0 tx=0).stop 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 -- 172.21.15.16:0/2058435551 shutdown_connections 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 --2- 172.21.15.16:0/2058435551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 0x7f50cc110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 --2- 172.21.15.16:0/2058435551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc10d8d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 --2- 172.21.15.16:0/2058435551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 0x7f50cc104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 -- 172.21.15.16:0/2058435551 >> 172.21.15.16:0/2058435551 conn(0x7f50cc079740 msgr2=0x7f50cc079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 -- 172.21.15.16:0/2058435551 shutdown_connections 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.481+0000 7f50d3053640 1 -- 172.21.15.16:0/2058435551 wait complete. 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 Processor -- start 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 -- start start 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 0x7f50cc19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 0x7f50cc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:52.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc113940 con 0x7f50cc105290 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc1137c0 con 0x7f50cc10de10 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50d3053640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc113ac0 con 0x7f50cc104860 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.482+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40526/0 (socket says 172.21.15.16:40526) 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 -- 172.21.15.16:0/852520773 learned_addr learned my addr 172.21.15.16:0/852520773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50d0def640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 0x7f50cc19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50d15f0640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 0x7f50cc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 -- 172.21.15.16:0/852520773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 msgr2=0x7f50cc19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 0x7f50cc19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 -- 172.21.15.16:0/852520773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 msgr2=0x7f50cc19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 0x7f50cc19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 -- 172.21.15.16:0/852520773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50cc1a0420 con 0x7f50cc105290 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50cbfff640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f50c0005940 tx=0x7f50c00040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:52.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c003f030 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50c0003a80 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50cc1a06b0 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.483+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0035070 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.484+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50cc1a6e70 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.484+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50cc06c1e0 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.486+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f50c0042030 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.487+0000 7f50c9ffb640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 0x7f50b40b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.487+0000 7f50d0def640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 0x7f50b40b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.487+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9017+0+0 (secure 0 0 0) 0x7f50c0108110 con 0x7f50cc105290 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.487+0000 7f50d0def640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 0x7f50b40b8a70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005fd0 tx=0x7f50bc005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.490+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f50c00ce630 con 0x7f50cc105290 2024-09-21T02:04:52.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.680+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50cc104c70 con 0x7f50cc105290 2024-09-21T02:04:52.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.682+0000 7f50c9ffb640 1 -- 172.21.15.16:0/852520773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+24530 (secure 0 0 0) 0x7f50cc104c70 con 0x7f50cc105290 2024-09-21T02:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:43.425075+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:04:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 msgr2=0x7f50b40b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 0x7f50b40b8a70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005fd0 tx=0x7f50bc005850 comp rx=0 tx=0).stop 2024-09-21T02:04:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 msgr2=0x7f50cc19aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f50c0005940 tx=0x7f50c00040e0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 shutdown_connections 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50b40b6560 0x7f50b40b8a70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc10de10 0x7f50cc19fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc105290 0x7f50cc19aa50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 --2- 172.21.15.16:0/852520773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50cc104860 0x7f50cc19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 >> 172.21.15.16:0/852520773 conn(0x7f50cc079740 msgr2=0x7f50cc0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 shutdown_connections 2024-09-21T02:04:52.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:52.684+0000 7f50d3053640 1 -- 172.21.15.16:0/852520773 wait complete. 2024-09-21T02:04:53.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:04:53.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]} : dispatch 2024-09-21T02:04:53.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]} : dispatch 2024-09-21T02:04:53.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:52 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/852520773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:53.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:04:53.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]} : dispatch 2024-09-21T02:04:53.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:52 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/852520773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]} : dispatch 2024-09-21T02:04:53.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:52 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/852520773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:04:53.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T02:04:34.359195+0000', 'modified': '2024-09-21T02:04:43.425228+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24710}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24710': {'gid': 24710, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/1116723582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1116723582}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1116723582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24710, 'qdb_cluster': [24710]} 2024-09-21T02:04:53.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T02:04:53.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:04:53.584 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-09-21T02:04:54.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:53 smithi016 ceph-mon[31245]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-21T02:04:54.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T02:04:54.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-21T02:04:54.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-09-21T02:04:54.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:53 smithi123 ceph-mon[33453]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 -- 172.21.15.16:0/3066175875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654108d20 msgr2=0x7f865410f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 --2- 172.21.15.16:0/3066175875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654108d20 0x7f865410f550 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ca0 tx=0x7f864c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 -- 172.21.15.16:0/3066175875 shutdown_connections 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 --2- 172.21.15.16:0/3066175875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654108d20 0x7f865410f550 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 --2- 172.21.15.16:0/3066175875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f86541087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 --2- 172.21.15.16:0/3066175875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654102ae0 0x7f8654102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 -- 172.21.15.16:0/3066175875 >> 172.21.15.16:0/3066175875 conn(0x7f8654079760 msgr2=0x7f8654079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 -- 172.21.15.16:0/3066175875 shutdown_connections 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.093+0000 7f8658d71640 1 -- 172.21.15.16:0/3066175875 wait complete. 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.094+0000 7f8658d71640 1 Processor -- start 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.094+0000 7f8658d71640 1 -- start start 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f865419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654108d20 0x7f865419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8654113990 con 0x7f8654102ae0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8654113810 con 0x7f86541083d0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8654113b10 con 0x7f8654108d20 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40546/0 (socket says 172.21.15.16:40546) 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 -- 172.21.15.16:0/756090627 learned_addr learned my addr 172.21.15.16:0/756090627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8652d9d640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654108d20 0x7f865419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8651d9b640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f865419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 -- 172.21.15.16:0/756090627 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654108d20 msgr2=0x7f865419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654108d20 0x7f865419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 -- 172.21.15.16:0/756090627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 msgr2=0x7f865419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f865419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 -- 172.21.15.16:0/756090627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86541a0250 con 0x7f8654102ae0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8651d9b640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f865419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f865259c640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f864400d770 tx=0x7f864400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f864b7fe640 1 -- 172.21.15.16:0/756090627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644017070 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f864b7fe640 1 -- 172.21.15.16:0/756090627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86440054b0 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f864b7fe640 1 -- 172.21.15.16:0/756090627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644004c60 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.095+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86541a0540 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.096+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86541a6d60 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.098+0000 7f865f2e5740 1 Processor -- start 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.098+0000 7f864b7fe640 1 -- 172.21.15.16:0/756090627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f8644004e00 con 0x7f8654102ae0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.098+0000 7f865f2e5740 1 -- start start 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.098+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.098+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d450a0 0x561f86114440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86114980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865f2e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f8608bc10 con 0x561f85d6df60 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865f2e5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f8607b040 con 0x561f85d450a0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865f2e5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f8607ee50 con 0x561f85d4d280 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40558/0 (socket says 172.21.15.16:40558) 2024-09-21T02:04:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 -- 172.21.15.16:0/3505803748 learned_addr learned my addr 172.21.15.16:0/3505803748 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865259c640 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d450a0 0x561f86114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8651d9b640 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86114980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 -- 172.21.15.16:0/3505803748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 msgr2=0x561f86114980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86114980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 -- 172.21.15.16:0/3505803748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d450a0 msgr2=0x561f86114440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d450a0 0x561f86114440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 -- 172.21.15.16:0/3505803748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f85dd9540 con 0x561f85d6df60 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f864b7fe640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 0x7f86400b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f865259c640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 0x7f86400b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ca0 tx=0x7f864c031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef0298376ae11bd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.099+0000 7f864b7fe640 1 -- 172.21.15.16:0/756090627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f86440e7230 con 0x7f8654102ae0 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f8648ff9640 1 -- 172.21.15.16:0/3505803748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c0317b0 con 0x561f85d6df60 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f8648ff9640 1 -- 172.21.15.16:0/3505803748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f864c036030 con 0x561f85d6df60 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f8648ff9640 1 -- 172.21.15.16:0/3505803748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c004530 con 0x561f85d6df60 2024-09-21T02:04:54.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f865f2e5740 1 -- 172.21.15.16:0/3505803748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 msgr2=0x561f86113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ca0 tx=0x7f864c031500 comp rx=0 tx=0).stop 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.100+0000 7f865259c640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 0x7f86400b8aa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x561f86112b80 tx=0x7f86440ead10 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- 172.21.15.16:0/3505803748 shutdown_connections 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d450a0 0x561f86114440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3505803748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d6df60 0x561f86113f00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- 172.21.15.16:0/3505803748 >> 172.21.15.16:0/3505803748 conn(0x561f85d62350 msgr2=0x561f86111df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- 172.21.15.16:0/3505803748 shutdown_connections 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- 172.21.15.16:0/3505803748 wait complete. 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- start start 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 Processor -- start 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 0x561f8611f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d6df60 0x561f8612b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f85edc5f0 con 0x561f85d4d280 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f86016010 con 0x561f85d6df60 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f86122500 con 0x561f85d450a0 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.101+0000 7f865f2e5740 10 client.0 _ll_register_callbacks cb 0x561f86075340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40562/0 (socket says 172.21.15.16:40562) 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 -- 172.21.15.16:0/2680997042 learned_addr learned my addr 172.21.15.16:0/2680997042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 0x561f8611f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 -- 172.21.15.16:0/2680997042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 msgr2=0x561f8611f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 0x561f8611f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 -- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d6df60 msgr2=0x561f8612b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d6df60 0x561f8612b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 -- 172.21.15.16:0/2680997042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f86043a00 con 0x561f85d4d280 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 0x561f8611f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f863c011810 tx=0x7f863c011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c91ee1a4f3e9a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c004460 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f863c00f9a0 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f864c036030 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f863c023050 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f865f2e5740 10 client.14955 fetch_fsmap learned FSMap version 17 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.102+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561f8612e4b0 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f864c036030 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f863c018020 con 0x561f85d4d280 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 10 client.14955 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 10 client.14955 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 20 client.14955 populate_metadata read hostname 'smithi016' 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d2c8e0 0x561f86131f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 4 client.14955 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:04:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f8608db60 con 0x561f85d2c8e0 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.103+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d2c8e0 0x561f86131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.104+0000 7f8639ffb640 10 client.14955 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.104+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d2c8e0 0x561f86131f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af53e4ae9c5120b server_cookie=68e3bad2da6f6c2e in_seq=0 out_seq=0 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f8639ffb640 1 -- 172.21.15.16:0/2680997042 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561f8608db60 con 0x561f85d2c8e0 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f8639ffb640 10 client.14955 handle_command_reply: tid=0 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 client.14955 shutdown 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 msgr2=0x561f8611ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f863c011810 tx=0x7f863c011cd0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 shutdown_connections 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d2c8e0 0x561f86131f50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f8639ffb640 0 client.14955 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d6df60 0x561f8612b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d4d280 0x561f8611ff20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2680997042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d450a0 0x561f8611f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 >> 172.21.15.16:0/2680997042 conn(0x561f85d62350 msgr2=0x561f861170e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 shutdown_connections 2024-09-21T02:04:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 1 -- 172.21.15.16:0/2680997042 wait complete. 2024-09-21T02:04:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.105+0000 7f865f2e5740 20 client.14955 trim_cache size 0 max 16384 2024-09-21T02:04:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.123+0000 7f865f2e5740 1 Processor -- start 2024-09-21T02:04:54.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.123+0000 7f865f2e5740 1 -- start start 2024-09-21T02:04:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8611ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f8612bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f86130290 con 0x561f85d450a0 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f86130960 con 0x561f85d2c8e0 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f865f2e5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f86131030 con 0x561f85d4d280 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40568/0 (socket says 172.21.15.16:40568) 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 -- 172.21.15.16:0/2822651469 learned_addr learned my addr 172.21.15.16:0/2822651469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f8612bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8652d9d640 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8611ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 -- 172.21.15.16:0/2822651469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 msgr2=0x561f8612bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f8612bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 -- 172.21.15.16:0/2822651469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 msgr2=0x561f8611ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8611ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 -- 172.21.15.16:0/2822651469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f86116d60 con 0x561f85d450a0 2024-09-21T02:04:54.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f8651d9b640 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f863c008510 tx=0x7f863c011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609112bfa6891527 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f863bfff640 1 -- 172.21.15.16:0/2822651469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c02b070 con 0x561f85d450a0 2024-09-21T02:04:54.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.124+0000 7f863bfff640 1 -- 172.21.15.16:0/2822651469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f863c00fe30 con 0x561f85d450a0 2024-09-21T02:04:54.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f863bfff640 1 -- 172.21.15.16:0/2822651469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c020040 con 0x561f85d450a0 2024-09-21T02:04:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 -- 172.21.15.16:0/2822651469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 msgr2=0x561f8612b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f863c008510 tx=0x7f863c011440 comp rx=0 tx=0).stop 2024-09-21T02:04:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 -- 172.21.15.16:0/2822651469 shutdown_connections 2024-09-21T02:04:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f8612bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f8612b760 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 --2- 172.21.15.16:0/2822651469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8611ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 -- 172.21.15.16:0/2822651469 >> 172.21.15.16:0/2822651469 conn(0x561f85d62350 msgr2=0x561f8611f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 -- 172.21.15.16:0/2822651469 shutdown_connections 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.125+0000 7f865f2e5740 1 -- 172.21.15.16:0/2822651469 wait complete. 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 -- start start 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 Processor -- start 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8613a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86134370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f86131dd0 con 0x561f85d450a0 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f86131700 con 0x561f85d2c8e0 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f861324a0 con 0x561f85d4d280 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f865f2e5740 10 client.0 _ll_register_callbacks cb 0x7f863c008760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:04:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f8651d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40584/0 (socket says 172.21.15.16:40584) 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f8651d9b640 1 -- 172.21.15.16:0/3030556579 learned_addr learned my addr 172.21.15.16:0/3030556579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.126+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8613a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 -- 172.21.15.16:0/3030556579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 msgr2=0x561f86134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86134370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 -- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 msgr2=0x561f8613a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8613a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 -- 172.21.15.16:0/3030556579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f86134ab0 con 0x561f85d450a0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8651d9b640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f863c0055c0 tx=0x7f863c0055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7900c381de45cdf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c02b040 con 0x561f85d450a0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f863c0035f0 con 0x561f85d450a0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f863c00fe30 con 0x561f85d450a0 2024-09-21T02:04:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f863c044050 con 0x561f85d450a0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f865f2e5740 10 client.14961 fetch_fsmap learned FSMap version 17 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561f8619d420 con 0x561f85d450a0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.127+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f863c00fe30 con 0x561f85d450a0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f863c018070 con 0x561f85d450a0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 10 client.14961 fetch_fsmap finished waiting for FSMap version 17 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 10 client.14961 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 20 client.14961 populate_metadata read hostname 'smithi016' 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d6df60 0x561f861a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 4 client.14961 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "status"}) -- 0x561f86089610 con 0x561f85d6df60 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d6df60 0x561f861a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f8652d9d640 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d6df60 0x561f861a0870 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf1a52b9b615c2a server_cookie=a814c9e1395337a3 in_seq=0 out_seq=0 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.128+0000 7f8639ffb640 10 client.14961 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8639ffb640 1 -- 172.21.15.16:0/3030556579 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561f86089610 con 0x561f85d6df60 2024-09-21T02:04:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8639ffb640 10 client.14961 handle_command_reply: tid=0 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 client.14961 shutdown 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 msgr2=0x561f86133e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f863c0055c0 tx=0x7f863c0055f0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 shutdown_connections 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f85d6df60 0x561f861a0870 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f85d4d280 0x561f86134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f85d450a0 0x561f86133e30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 --2- 172.21.15.16:0/3030556579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f85d2c8e0 0x561f8613a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 >> 172.21.15.16:0/3030556579 conn(0x561f85d62350 msgr2=0x561f8611a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8639ffb640 0 client.14961 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 shutdown_connections 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 1 -- 172.21.15.16:0/3030556579 wait complete. 2024-09-21T02:04:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f865f2e5740 20 client.14961 trim_cache size 0 max 16384 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24710, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 19.482153363999998, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 17, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 84, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 81, 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 28.677525636999999 2024-09-21T02:04:54.136 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:04:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 msgr2=0x7f86400b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 0x7f86400b8aa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x561f86112b80 tx=0x7f86440ead10 comp rx=0 tx=0).stop 2024-09-21T02:04:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 msgr2=0x7f865419a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.129+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f864400d770 tx=0x7f864400dc30 comp rx=0 tx=0).stop 2024-09-21T02:04:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 shutdown_connections 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f86400b6590 0x7f86400b8aa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654108d20 0x7f865419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86541083d0 0x7f865419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 --2- 172.21.15.16:0/756090627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8654102ae0 0x7f865419a3a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 >> 172.21.15.16:0/756090627 conn(0x7f8654079760 msgr2=0x7f86541044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 shutdown_connections 2024-09-21T02:04:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:54.130+0000 7f8658d71640 1 -- 172.21.15.16:0/756090627 wait complete. 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-09-21T02:04:54.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:53 smithi161 ceph-mon[34061]: osdmap e83: 12 total, 12 up, 12 in 2024-09-21T02:04:54.913 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T02:04:54.914 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:54.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T02:04:55.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:54 smithi123 ceph-mon[33453]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:04:55.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:54 smithi016 ceph-mon[31245]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:04:55.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:54 smithi161 ceph-mon[34061]: osdmap e84: 12 total, 12 up, 12 in 2024-09-21T02:04:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.652+0000 7f9e1a809640 1 -- 172.21.15.16:0/4114059513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1410de60 msgr2=0x7f9e14110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.652+0000 7f9e1a809640 1 --2- 172.21.15.16:0/4114059513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1410de60 0x7f9e14110330 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009ca0 tx=0x7f9e0c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 -- 172.21.15.16:0/4114059513 shutdown_connections 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 --2- 172.21.15.16:0/4114059513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e1410de60 0x7f9e14110330 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 --2- 172.21.15.16:0/4114059513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141052e0 0x7f9e1410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 --2- 172.21.15.16:0/4114059513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e141048b0 0x7f9e14104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 -- 172.21.15.16:0/4114059513 >> 172.21.15.16:0/4114059513 conn(0x7f9e14079750 msgr2=0x7f9e14079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 -- 172.21.15.16:0/4114059513 shutdown_connections 2024-09-21T02:04:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 -- 172.21.15.16:0/4114059513 wait complete. 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 Processor -- start 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.653+0000 7f9e1a809640 1 -- start start 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141048b0 0x7f9e1419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e1410de60 0x7f9e141a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e141139a0 con 0x7f9e141052e0 2024-09-21T02:04:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14113820 con 0x7f9e1410de60 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e1a809640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14113b20 con 0x7f9e141048b0 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40598/0 (socket says 172.21.15.16:40598) 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 -- 172.21.15.16:0/1632562887 learned_addr learned my addr 172.21.15.16:0/1632562887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e18da6640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e1410de60 0x7f9e141a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e13fff640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141048b0 0x7f9e1419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 -- 172.21.15.16:0/1632562887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141048b0 msgr2=0x7f9e1419c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141048b0 0x7f9e1419c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 -- 172.21.15.16:0/1632562887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e1410de60 msgr2=0x7f9e141a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e1410de60 0x7f9e141a1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.654+0000 7f9e137fe640 1 -- 172.21.15.16:0/1632562887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e141a2500 con 0x7f9e141052e0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e137fe640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800d770 tx=0x7f9e0800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08017070 con 0x7f9e141052e0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e08009c30 con 0x7f9e141052e0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0800f620 con 0x7f9e141052e0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e141a27f0 con 0x7f9e141052e0 2024-09-21T02:04:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.655+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e141a8f50 con 0x7f9e141052e0 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.656+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1406c1e0 con 0x7f9e141052e0 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.657+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f9e0801a070 con 0x7f9e141052e0 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.658+0000 7f9e117fa640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 0x7f9dfc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.658+0000 7f9e13fff640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 0x7f9dfc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.658+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f9e080e7140 con 0x7f9e141052e0 2024-09-21T02:04:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.659+0000 7f9e13fff640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 0x7f9dfc0b8a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406c8e0 tx=0x7f9e04005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:04:55.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.661+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9e08009de0 con 0x7f9e141052e0 2024-09-21T02:04:55.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.892+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9e14104cc0 con 0x7f9e141052e0 2024-09-21T02:04:55.895 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:55 smithi016 ceph-mon[31245]: pgmap v283: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:55.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.893+0000 7f9e117fa640 1 -- 172.21.15.16:0/1632562887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f9e080b1980 con 0x7f9e141052e0 2024-09-21T02:04:55.895 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T02:04:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.896+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 msgr2=0x7f9dfc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:55.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.896+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 0x7f9dfc0b8a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9e1406c8e0 tx=0x7f9e04005f90 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.896+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 msgr2=0x7f9e1419cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.896+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800d770 tx=0x7f9e0800dc30 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 shutdown_connections 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9dfc0b6560 0x7f9dfc0b8a70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e1410de60 0x7f9e141a1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e141052e0 0x7f9e1419cb30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 --2- 172.21.15.16:0/1632562887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e141048b0 0x7f9e1419c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 >> 172.21.15.16:0/1632562887 conn(0x7f9e14079750 msgr2=0x7f9e140ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 shutdown_connections 2024-09-21T02:04:55.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:55.897+0000 7f9e1a809640 1 -- 172.21.15.16:0/1632562887 wait complete. 2024-09-21T02:04:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:55 smithi123 ceph-mon[33453]: pgmap v283: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:56.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:55 smithi161 ceph-mon[34061]: pgmap v283: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:56.585 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-21T02:04:56.585 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T02:04:56.585 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-09-21T02:04:56.585 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-21T02:04:56.585 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:56.586 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-21T02:04:56.586 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T02:04:56.586 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T02:04:56.586 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T02:04:56.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:56.586 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:04:56.604 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42230sec preferred_lft 42230sec 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:04:56.605 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-09-21T02:04:56.606 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T02:04:56.606 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:04:56.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:04:56.809 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:56 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1632562887' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:04:56.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:56.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:56.844 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T02:04:56.869 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-09-21T02:04:56.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:56.871 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-09-21T02:04:56.939 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-09-21T02:04:56.939 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T02:04:56.940 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T02:04:56.940 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T02:04:56.940 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T02:04:56.940 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:04:57.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:56 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1632562887' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:04:57.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:04:57.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:56 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1632562887' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:04:57.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:57.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:57.261 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-21T02:04:57.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:57.278 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-09-21T02:04:57.334 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:04:57.334 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:04:57.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:57.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-21T02:04:57.334 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:04:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:04:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:57.455 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-21T02:04:57.455 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:04:57.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:04:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:57.587 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:04:57.587 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:04:57.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-09-21T02:04:57.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T02:04:57.587 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:04:57.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:04:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:57.669 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-21T02:04:57.669 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:04:57.670 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-21T02:04:57.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:04:57.685 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:57.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:04:57.814 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:04:57.870 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:04:57.870 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:04:57.871 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:04:57.872 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:57.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:57.873 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:04:57.926 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T02:04:57.926 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-21T02:04:57.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:04:57.995 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:04:58.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:57 smithi123 ceph-mon[33453]: pgmap v284: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:58.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:57 smithi016 ceph-mon[31245]: pgmap v284: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:04:58.043 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:04:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:04:58.045 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:58.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:58.046 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:04:58.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2024-09-21T02:04:58.126+0000 7f147dceb480 -1 init, newargv = 0x7f146c004260 newargc=15 2024-09-21T02:04:58.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[77447]: starting ceph client 2024-09-21T02:04:58.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[77447]: starting fuse 2024-09-21T02:04:58.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:57 smithi161 ceph-mon[34061]: pgmap v284: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:04:59.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:04:59.235 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:04:59.295 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:04:59.296 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:04:59.297 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:04:59.298 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:59.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:59.299 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:04:59.358 INFO:teuthology.orchestra.run.smithi016.stdout:108 2024-09-21T02:04:59.358 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [108] 2024-09-21T02:04:59.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-09-21T02:04:59.358 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-09-21T02:04:59.358 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-09-21T02:04:59.358 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-09-21T02:04:59.358 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-09-21T02:04:59.359 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-09-21T02:04:59.360 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-09-21T02:04:59.360 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T02:04:59.360 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:04:59.360 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2024-09-21T02:04:59.360 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-09-21T02:04:59.486 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.77447.asok 2024-09-21T02:04:59.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:59.496 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.77447.asok 2024-09-21T02:04:59.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:59.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.77447.asok status 2024-09-21T02:04:59.644 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "77447", 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-09-21T02:04:59.645 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 14973, 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 14973 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 720237585 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:04:59.646 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 720237585 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.14973 192.168.0.1:0/720237585", 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/720237585", 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 17, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 84, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-09-21T02:04:59.647 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:04:59.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:59.654 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-09-21T02:04:59.654 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:59.670 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-09-21T02:04:59.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:59.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:04:59.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:04:59.738 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-21T02:04:59.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-21T02:04:59.852 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2024-09-21T02:04:59.852 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-21T02:04:59.852 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:04:59.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:04:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:04:59.858 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:04:59.859 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:04:59.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:05:00.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:04:59 smithi123 ceph-mon[33453]: pgmap v285: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:04:59 smithi016 ceph-mon[31245]: pgmap v285: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:00.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:04:59 smithi161 ceph-mon[34061]: pgmap v285: 129 pgs: 2 activating, 1 peering, 126 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:00.174 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 -- 172.21.15.16:0/2449074337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0103bb0 msgr2=0x7f93f0103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2449074337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0103bb0 0x7f93f0103fc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ca0 tx=0x7f93e002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 -- 172.21.15.16:0/2449074337 shutdown_connections 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2449074337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0109e40 0x7f93f0110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2449074337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 0x7f93f0109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2449074337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0103bb0 0x7f93f0103fc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 -- 172.21.15.16:0/2449074337 >> 172.21.15.16:0/2449074337 conn(0x7f93f00ff600 msgr2=0x7f93f0101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 -- 172.21.15.16:0/2449074337 shutdown_connections 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.677+0000 7f93f5c89640 1 -- 172.21.15.16:0/2449074337 wait complete. 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 Processor -- start 2024-09-21T02:05:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 -- start start 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0103bb0 0x7f93f019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 0x7f93f019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93f0113a00 con 0x7f93f0109e40 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93f0113880 con 0x7f93f0103bb0 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93f5c89640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f93f0113b80 con 0x7f93f01094f0 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.678+0000 7f93effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43520/0 (socket says 172.21.15.16:43520) 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 0x7f93f019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 -- 172.21.15.16:0/2098653610 learned_addr learned my addr 172.21.15.16:0/2098653610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:00.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93ef7fe640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0103bb0 0x7f93f019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 -- 172.21.15.16:0/2098653610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 msgr2=0x7f93f019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 0x7f93f019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 -- 172.21.15.16:0/2098653610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0103bb0 msgr2=0x7f93f019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0103bb0 0x7f93f019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 -- 172.21.15.16:0/2098653610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f01a0380 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93effff640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c970 tx=0x7f93e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8007bb0 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f01a0670 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93e8007d50 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8004d50 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.679+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f01a6dd0 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.680+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f0103fc0 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.682+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f93e8005000 con 0x7f93f0109e40 2024-09-21T02:05:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.683+0000 7f93ecff9640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 0x7f93dc0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.683+0000 7f93ef7fe640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 0x7f93dc0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.683+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f93e80e7820 con 0x7f93f0109e40 2024-09-21T02:05:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.684+0000 7f93ef7fe640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 0x7f93dc0b8a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f80 tx=0x7f93e00093f0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.687+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f93e8014030 con 0x7f93f0109e40 2024-09-21T02:05:00.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.889+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f93f019b540 con 0x7f93f0109e40 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.890+0000 7f93ecff9640 1 -- 172.21.15.16:0/2098653610 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v84) ==== 81+0+888 (secure 0 0 0) 0x7f93f019b540 con 0x7f93f0109e40 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3095458517 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2441291809 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:00.891 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3677616942 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4075907232 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2368126584 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3423016963 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3586667721 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429917716 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3010596393 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:00.892 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-09-21T02:05:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 msgr2=0x7f93dc0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 0x7f93dc0b8a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f80 tx=0x7f93e00093f0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 msgr2=0x7f93f019fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c970 tx=0x7f93e800ce30 comp rx=0 tx=0).stop 2024-09-21T02:05:00.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 shutdown_connections 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f93dc0b6560 0x7f93dc0b8a70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f0109e40 0x7f93f019fc50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93f01094f0 0x7f93f019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 --2- 172.21.15.16:0/2098653610 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f0103bb0 0x7f93f019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.892+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 >> 172.21.15.16:0/2098653610 conn(0x7f93f00ff600 msgr2=0x7f93f0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.893+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 shutdown_connections 2024-09-21T02:05:00.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:00.893+0000 7f93f5c89640 1 -- 172.21.15.16:0/2098653610 wait complete. 2024-09-21T02:05:01.821 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:05:01.821 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:05:01.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:05:02.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:01 smithi123 ceph-mon[33453]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:02.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:01 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2098653610' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:02.148 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:01 smithi016 ceph-mon[31245]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:01 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2098653610' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:02.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:01 smithi161 ceph-mon[34061]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:02.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:01 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2098653610' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:02.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 -- 172.21.15.16:0/1436829072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 msgr2=0x7f753c103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:02.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 --2- 172.21.15.16:0/1436829072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c103f70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f752c009900 tx=0x7f752c02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 -- 172.21.15.16:0/1436829072 shutdown_connections 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 --2- 172.21.15.16:0/1436829072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c109df0 0x7f753c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 --2- 172.21.15.16:0/1436829072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c1094a0 0x7f753c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 --2- 172.21.15.16:0/1436829072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c103f70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 -- 172.21.15.16:0/1436829072 >> 172.21.15.16:0/1436829072 conn(0x7f753c07b850 msgr2=0x7f753c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 -- 172.21.15.16:0/1436829072 shutdown_connections 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.634+0000 7f7541398640 1 -- 172.21.15.16:0/1436829072 wait complete. 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 Processor -- start 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 -- start start 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 0x7f753c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c109df0 0x7f753c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f753c113960 con 0x7f753c103b60 2024-09-21T02:05:02.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1137e0 con 0x7f753c1094a0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f7541398640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f753c113ae0 con 0x7f753c109df0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f753affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f753affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43544/0 (socket says 172.21.15.16:43544) 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f753affd640 1 -- 172.21.15.16:0/2383114123 learned_addr learned my addr 172.21.15.16:0/2383114123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.635+0000 7f753b7fe640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c109df0 0x7f753c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753a7fc640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 0x7f753c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 -- 172.21.15.16:0/2383114123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c109df0 msgr2=0x7f753c19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c109df0 0x7f753c19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 -- 172.21.15.16:0/2383114123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 msgr2=0x7f753c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 0x7f753c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 -- 172.21.15.16:0/2383114123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c1a0240 con 0x7f753c103b60 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753a7fc640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 0x7f753c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f753affd640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f752c009230 tx=0x7f752c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:02.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c03f030 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c1a04d0 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f752c003a80 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c035070 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.636+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a09b0 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.637+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f753c103f70 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.639+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f752c042030 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.639+0000 7f7523fff640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 0x7f75240b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.640+0000 7f753a7fc640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 0x7f75240b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.640+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f752c107f80 con 0x7f753c103b60 2024-09-21T02:05:02.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.640+0000 7f753a7fc640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 0x7f75240b8aa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7530005d00 tx=0x7f7530005ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:02.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.644+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f752c0ce3e0 con 0x7f753c103b60 2024-09-21T02:05:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.848+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f753c02e640 con 0x7f753c103b60 2024-09-21T02:05:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.848+0000 7f7523fff640 1 -- 172.21.15.16:0/2383114123 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v84) ==== 81+0+888 (secure 0 0 0) 0x7f752c0d27c0 con 0x7f753c103b60 2024-09-21T02:05:02.849 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3095458517 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:02.849 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2441291809 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3677616942 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4075907232 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2368126584 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3423016963 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3586667721 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429917716 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3010596393 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:02.850 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:05:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-09-21T02:05:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 msgr2=0x7f75240b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 0x7f75240b8aa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7530005d00 tx=0x7f7530005ed0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 msgr2=0x7f753c19a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f752c009230 tx=0x7f752c004270 comp rx=0 tx=0).stop 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 shutdown_connections 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.850+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f75240b6590 0x7f75240b8aa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f753c109df0 0x7f753c19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f753c1094a0 0x7f753c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 --2- 172.21.15.16:0/2383114123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f753c103b60 0x7f753c19a390 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 >> 172.21.15.16:0/2383114123 conn(0x7f753c07b850 msgr2=0x7f753c1075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 shutdown_connections 2024-09-21T02:05:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:02.851+0000 7f7541398640 1 -- 172.21.15.16:0/2383114123 wait complete. 2024-09-21T02:05:03.653 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-09-21T02:05:03.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:05:03.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-21T02:05:03.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[77447]: fuse finished with error 0 and tester_r 0 2024-09-21T02:05:03.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T02:05:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:03 smithi016 ceph-mon[31245]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2383114123' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:04.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:03 smithi123 ceph-mon[33453]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:04.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2383114123' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:03 smithi161 ceph-mon[34061]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2383114123' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:05:06.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:05 smithi123 ceph-mon[33453]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 398 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:05 smithi016 ceph-mon[31245]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 398 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:06.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:05 smithi161 ceph-mon[34061]: pgmap v288: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 398 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:08.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:07 smithi123 ceph-mon[33453]: pgmap v289: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:07 smithi016 ceph-mon[31245]: pgmap v289: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:08.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:07 smithi161 ceph-mon[34061]: pgmap v289: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T02:05:09.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:05:09.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:05:09.790 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-21T02:05:09.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:05:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:09 smithi016 ceph-mon[31245]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:05:10.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:09 smithi161 ceph-mon[34061]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:05:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:09 smithi123 ceph-mon[33453]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-21T02:05:10.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 -- 172.21.15.16:0/3551678001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c4102ae0 msgr2=0x7f00c4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:10.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/3551678001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c4102ae0 0x7f00c4102ef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009900 tx=0x7f00b402a410 comp rx=0 tx=0).stop 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 -- 172.21.15.16:0/3551678001 shutdown_connections 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/3551678001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 0x7f00c410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/3551678001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c41083d0 0x7f00c41087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/3551678001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c4102ae0 0x7f00c4102ef0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 -- 172.21.15.16:0/3551678001 >> 172.21.15.16:0/3551678001 conn(0x7f00c4079760 msgr2=0x7f00c4079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 -- 172.21.15.16:0/3551678001 shutdown_connections 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.632+0000 7f00c8d9e640 1 -- 172.21.15.16:0/3551678001 wait complete. 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.633+0000 7f00c8d9e640 1 Processor -- start 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.633+0000 7f00c8d9e640 1 -- start start 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c4102ae0 0x7f00c419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:10.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 0x7f00c419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4113960 con 0x7f00c41083d0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41137e0 con 0x7f00c4102ae0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c8d9e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4113ae0 con 0x7f00c4108d20 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58778/0 (socket says 172.21.15.16:58778) 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 -- 172.21.15.16:0/4175860793 learned_addr learned my addr 172.21.15.16:0/4175860793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c2d9d640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 0x7f00c419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c259c640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c4102ae0 0x7f00c419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 -- 172.21.15.16:0/4175860793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 msgr2=0x7f00c419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 0x7f00c419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 -- 172.21.15.16:0/4175860793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c4102ae0 msgr2=0x7f00c419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c4102ae0 0x7f00c419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 -- 172.21.15.16:0/4175860793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41a02a0 con 0x7f00c41083d0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00c1d9b640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00b800e9c0 tx=0x7f00b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:10.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.634+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b800cd80 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.635+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c41a0590 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.635+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f00b80054b0 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.635+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8010430 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.635+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41a6cf0 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.636+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c4102ef0 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.638+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f00b8004f20 con 0x7f00c41083d0 2024-09-21T02:05:10.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.639+0000 7f00b37fe640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 0x7f00ac0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.640+0000 7f00c259c640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 0x7f00ac0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.640+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f00b80e7e10 con 0x7f00c41083d0 2024-09-21T02:05:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.640+0000 7f00c259c640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 0x7f00ac0b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b4009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.644+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f00b80ae270 con 0x7f00c41083d0 2024-09-21T02:05:10.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.838+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00c402e640 con 0x7f00c41083d0 2024-09-21T02:05:10.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.840+0000 7f00b37fe640 1 -- 172.21.15.16:0/4175860793 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25057 (secure 0 0 0) 0x7f00b80b2650 con 0x7f00c41083d0 2024-09-21T02:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:53.733185+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 msgr2=0x7f00ac0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 0x7f00ac0b8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b4009680 comp rx=0 tx=0).stop 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 msgr2=0x7f00c419a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00b800e9c0 tx=0x7f00b800ee80 comp rx=0 tx=0).stop 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 shutdown_connections 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f00ac0b6560 0x7f00ac0b8a70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c4108d20 0x7f00c419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00c41083d0 0x7f00c419a8d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 --2- 172.21.15.16:0/4175860793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c4102ae0 0x7f00c419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:10.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 >> 172.21.15.16:0/4175860793 conn(0x7f00c4079760 msgr2=0x7f00c4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:10.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 shutdown_connections 2024-09-21T02:05:10.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:10.842+0000 7f00c8d9e640 1 -- 172.21.15.16:0/4175860793 wait complete. 2024-09-21T02:05:11.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:05:11.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:11 smithi016 ceph-mon[31245]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:05:11.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4175860793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:11 smithi161 ceph-mon[34061]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:05:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4175860793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:11 smithi123 ceph-mon[33453]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-21T02:05:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4175860793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:12.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/3258942355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0102a60 msgr2=0x7fa5e0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:12.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/3258942355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0102a60 0x7fa5e0102e70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009900 tx=0x7fa5d002a410 comp rx=0 tx=0).stop 2024-09-21T02:05:12.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/3258942355 shutdown_connections 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/3258942355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e0108cf0 0x7fa5e010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/3258942355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e01083a0 0x7fa5e01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/3258942355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0102a60 0x7fa5e0102e70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/3258942355 >> 172.21.15.16:0/3258942355 conn(0x7fa5e0079730 msgr2=0x7fa5e0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/3258942355 shutdown_connections 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.375+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/3258942355 wait complete. 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.376+0000 7fa5e5dd3640 1 Processor -- start 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.376+0000 7fa5e5dd3640 1 -- start start 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e0102a60 0x7fa5e019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e01083a0 0x7fa5e019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e0112860 con 0x7fa5e0108cf0 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e01126e0 con 0x7fa5e01083a0 2024-09-21T02:05:12.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5e5dd3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e01129e0 con 0x7fa5e0102a60 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58796/0 (socket says 172.21.15.16:58796) 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 -- 172.21.15.16:0/4075922655 learned_addr learned my addr 172.21.15.16:0/4075922655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5deffd640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e01083a0 0x7fa5e019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5df7fe640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e0102a60 0x7fa5e019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 -- 172.21.15.16:0/4075922655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e0102a60 msgr2=0x7fa5e019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e0102a60 0x7fa5e019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 -- 172.21.15.16:0/4075922655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e01083a0 msgr2=0x7fa5e019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e01083a0 0x7fa5e019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 -- 172.21.15.16:0/4075922655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e01a02d0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.377+0000 7fa5dffff640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800eaa0 tx=0x7fa5d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.378+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d800cdb0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.378+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e01a05c0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.378+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5d8005020 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.378+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d80116d0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.378+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e01a6d20 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.379+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e0102e70 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.380+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fa5d80051d0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.381+0000 7fa5dcff9640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 0x7fa5c80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.382+0000 7fa5df7fe640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 0x7fa5c80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.382+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fa5d80e78d0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.382+0000 7fa5df7fe640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 0x7fa5c80b8a70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006010 tx=0x7fa5d0000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.386+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa5d8016030 con 0x7fa5e0108cf0 2024-09-21T02:05:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.574+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5e01a7110 con 0x7fa5e0108cf0 2024-09-21T02:05:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.576+0000 7fa5dcff9640 1 -- 172.21.15.16:0/4075922655 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25057 (secure 0 0 0) 0x7fa5d80b20b0 con 0x7fa5e0108cf0 2024-09-21T02:05:12.577 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:12.577 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:53.733185+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 msgr2=0x7fa5c80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 0x7fa5c80b8a70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006010 tx=0x7fa5d0000bc0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 msgr2=0x7fa5e019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800eaa0 tx=0x7fa5d800ef60 comp rx=0 tx=0).stop 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 shutdown_connections 2024-09-21T02:05:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5c80b6560 0x7fa5c80b8a70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5e0108cf0 0x7fa5e019fba0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5e01083a0 0x7fa5e019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 --2- 172.21.15.16:0/4075922655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e0102a60 0x7fa5e019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 >> 172.21.15.16:0/4075922655 conn(0x7fa5e0079730 msgr2=0x7fa5e0101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 shutdown_connections 2024-09-21T02:05:12.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:12.578+0000 7fa5e5dd3640 1 -- 172.21.15.16:0/4075922655 wait complete. 2024-09-21T02:05:12.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:12 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4075922655' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:12 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4075922655' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:13.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:12 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4075922655' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:13.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs set cephfs standby_count_wanted 0 2024-09-21T02:05:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:14.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:13 smithi016 ceph-mon[31245]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 -- 172.21.15.16:0/3975969500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce0103bc0 msgr2=0x7fbce0103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 --2- 172.21.15.16:0/3975969500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce0103bc0 0x7fbce0103fd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009ca0 tx=0x7fbcd802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 -- 172.21.15.16:0/3975969500 shutdown_connections 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 --2- 172.21.15.16:0/3975969500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 0x7fbce0110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 --2- 172.21.15.16:0/3975969500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce01094b0 0x7fbce01098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 --2- 172.21.15.16:0/3975969500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce0103bc0 0x7fbce0103fd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 -- 172.21.15.16:0/3975969500 >> 172.21.15.16:0/3975969500 conn(0x7fbce007b910 msgr2=0x7fbce007bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 -- 172.21.15.16:0/3975969500 shutdown_connections 2024-09-21T02:05:14.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.122+0000 7fbce5096640 1 -- 172.21.15.16:0/3975969500 wait complete. 2024-09-21T02:05:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 Processor -- start 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 -- start start 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 0x7fbce0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbce01139c0 con 0x7fbce01094b0 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0113840 con 0x7fbce0103bc0 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.123+0000 7fbce5096640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0113b40 con 0x7fbce0109e00 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcded9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcded9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:36124/0 (socket says 172.21.15.16:36124) 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58812/0 (socket says 172.21.15.16:58812) 2024-09-21T02:05:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcded9d640 1 -- 172.21.15.16:0/3235507639 learned_addr learned my addr 172.21.15.16:0/3235507639 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcdf59e640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 0x7fbce0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 -- 172.21.15.16:0/3235507639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 msgr2=0x7fbce0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 0x7fbce0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 -- 172.21.15.16:0/3235507639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 msgr2=0x7fbce0075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 -- 172.21.15.16:0/3235507639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce01a8600 con 0x7fbce01094b0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcded9d640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcde59c640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbce006bb30 tx=0x7fbcc800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8007bb0 con 0x7fbce01094b0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbcc8007d50 con 0x7fbce01094b0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce01a88f0 con 0x7fbce01094b0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.124+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8004d10 con 0x7fbce01094b0 2024-09-21T02:05:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.125+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01a8e80 con 0x7fbce01094b0 2024-09-21T02:05:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.126+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce0103fd0 con 0x7fbce01094b0 2024-09-21T02:05:14.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.127+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fbcc8004eb0 con 0x7fbce01094b0 2024-09-21T02:05:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.128+0000 7fbcd7fff640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 0x7fbccc0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.128+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fbcc80e7620 con 0x7fbce01094b0 2024-09-21T02:05:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.128+0000 7fbcded9d640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 0x7fbccc0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.129+0000 7fbcded9d640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 0x7fbccc0b8af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8031040 tx=0x7fbcd8009460 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.133+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbcc8014030 con 0x7fbce01094b0 2024-09-21T02:05:14.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:13 smithi161 ceph-mon[34061]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:13 smithi123 ceph-mon[33453]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.358+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fbce002e640 con 0x7fbce01094b0 2024-09-21T02:05:14.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.789+0000 7fbcd7fff640 1 -- 172.21.15.16:0/3235507639 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fbcc80b1e60 con 0x7fbce01094b0 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 msgr2=0x7fbccc0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 0x7fbccc0b8af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8031040 tx=0x7fbcd8009460 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 msgr2=0x7fbce0075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbce006bb30 tx=0x7fbcc800cdd0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 shutdown_connections 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbccc0b65e0 0x7fbccc0b8af0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce0109e00 0x7fbce0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce01094b0 0x7fbce0075c50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 --2- 172.21.15.16:0/3235507639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0103bc0 0x7fbce0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 >> 172.21.15.16:0/3235507639 conn(0x7fbce007b910 msgr2=0x7fbce0107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 shutdown_connections 2024-09-21T02:05:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:14.792+0000 7fbce5096640 1 -- 172.21.15.16:0/3235507639 wait complete. 2024-09-21T02:05:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:14 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:05:15.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:14 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:05:15.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:14 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} : dispatch 2024-09-21T02:05:15.585 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-09-21T02:05:15.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs set cephfs max_mds 5 2024-09-21T02:05:15.847 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:15 smithi016 ceph-mon[31245]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:15 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:05:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:15 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:16.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:15 smithi161 ceph-mon[34061]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:16.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:15 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:05:16.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:15 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:15 smithi123 ceph-mon[33453]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-09-21T02:05:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:15 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3235507639' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-21T02:05:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:15 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.333+0000 7ff760a9b640 1 -- 172.21.15.16:0/1293041970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c105b30 msgr2=0x7ff75c10f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.333+0000 7ff760a9b640 1 --2- 172.21.15.16:0/1293041970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c105b30 0x7ff75c10f580 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff754009ca0 tx=0x7ff75402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 -- 172.21.15.16:0/1293041970 shutdown_connections 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 --2- 172.21.15.16:0/1293041970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c105b30 0x7ff75c10f580 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 --2- 172.21.15.16:0/1293041970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c1051e0 0x7ff75c1055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 --2- 172.21.15.16:0/1293041970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 -- 172.21.15.16:0/1293041970 >> 172.21.15.16:0/1293041970 conn(0x7ff75c079f90 msgr2=0x7ff75c07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 -- 172.21.15.16:0/1293041970 shutdown_connections 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.334+0000 7ff760a9b640 1 -- 172.21.15.16:0/1293041970 wait complete. 2024-09-21T02:05:16.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 Processor -- start 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 -- start start 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c105b30 0x7ff75c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c1128d0 con 0x7ff75c1051e0 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c112750 con 0x7ff75c105b30 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff760a9b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c112a50 con 0x7ff75c103210 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58840/0 (socket says 172.21.15.16:58840) 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 -- 172.21.15.16:0/3860095988 learned_addr learned my addr 172.21.15.16:0/3860095988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff75ad9d640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c105b30 0x7ff75c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff75a59c640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 -- 172.21.15.16:0/3860095988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 msgr2=0x7ff75c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c19a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 -- 172.21.15.16:0/3860095988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c105b30 msgr2=0x7ff75c19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c105b30 0x7ff75c19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff759d9b640 1 -- 172.21.15.16:0/3860095988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c1a0360 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.335+0000 7ff75a59c640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff759d9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff75000d770 tx=0x7ff75000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750017070 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c1a0650 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff750009c30 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75000f620 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.336+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75c1a6db0 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.337+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75c103620 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.339+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7ff75001a070 con 0x7ff75c1051e0 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.340+0000 7ff74b7fe640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 0x7ff7440b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:16.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.340+0000 7ff75a59c640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 0x7ff7440b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:16.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.340+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7ff7500e71f0 con 0x7ff75c1051e0 2024-09-21T02:05:16.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.340+0000 7ff75a59c640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 0x7ff7440b8a70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0058c0 tx=0x7ff74c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:16.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.343+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff750009de0 con 0x7ff75c1051e0 2024-09-21T02:05:16.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.570+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7ff75c02e640 con 0x7ff75c1051e0 2024-09-21T02:05:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.799+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3860095988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7ff7500ad650 con 0x7ff75c1051e0 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 msgr2=0x7ff7440b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 0x7ff7440b8a70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0058c0 tx=0x7ff74c005850 comp rx=0 tx=0).stop 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 msgr2=0x7ff75c19a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff75000d770 tx=0x7ff75000dc30 comp rx=0 tx=0).stop 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 shutdown_connections 2024-09-21T02:05:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7440b6560 0x7ff7440b8a70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff75c105b30 0x7ff75c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff75c1051e0 0x7ff75c19a990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 --2- 172.21.15.16:0/3860095988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff75c103210 0x7ff75c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.801+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 >> 172.21.15.16:0/3860095988 conn(0x7ff75c079f90 msgr2=0x7ff75c106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.802+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 shutdown_connections 2024-09-21T02:05:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:16.802+0000 7ff760a9b640 1 -- 172.21.15.16:0/3860095988 wait complete. 2024-09-21T02:05:16.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:16 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:05:17.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:16 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:05:17.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:16 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} : dispatch 2024-09-21T02:05:17.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs set cephfs allow_standby_replay true 2024-09-21T02:05:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-21T02:05:18.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:17 smithi016 ceph-mon[31245]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-21T02:05:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:17 smithi161 ceph-mon[34061]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-21T02:05:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-21T02:05:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T02:05:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3860095988' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-21T02:05:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-21T02:05:18.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-21T02:05:18.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-21T02:05:18.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:17 smithi123 ceph-mon[33453]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.312+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/2064112796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 msgr2=0x7f5c88102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.312+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/2064112796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c88102e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009900 tx=0x7f5c7802a410 comp rx=0 tx=0).stop 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/2064112796 shutdown_connections 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/2064112796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 0x7f5c8810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/2064112796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 0x7f5c881087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/2064112796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c88102e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/2064112796 >> 172.21.15.16:0/2064112796 conn(0x7f5c88079730 msgr2=0x7f5c88079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/2064112796 shutdown_connections 2024-09-21T02:05:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.313+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/2064112796 wait complete. 2024-09-21T02:05:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 Processor -- start 2024-09-21T02:05:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 -- start start 2024-09-21T02:05:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 0x7f5c8819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 0x7f5c8819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c88113930 con 0x7f5c88102a60 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c881137b0 con 0x7f5c881083a0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8cc3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5c88113ab0 con 0x7f5c88108cf0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39690/0 (socket says 172.21.15.16:39690) 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c8659c640 1 -- 172.21.15.16:0/1481755947 learned_addr learned my addr 172.21.15.16:0/1481755947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c85d9b640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 0x7f5c8819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.314+0000 7f5c86d9d640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 0x7f5c8819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 -- 172.21.15.16:0/1481755947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 msgr2=0x7f5c8819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 0x7f5c8819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 -- 172.21.15.16:0/1481755947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 msgr2=0x7f5c8819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 0x7f5c8819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 -- 172.21.15.16:0/1481755947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c881a02b0 con 0x7f5c88102a60 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8659c640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009230 tx=0x7f5c78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7803f030 con 0x7f5c88102a60 2024-09-21T02:05:18.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c881a0540 con 0x7f5c88102a60 2024-09-21T02:05:18.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5c78003a80 con 0x7f5c88102a60 2024-09-21T02:05:18.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c881a0a20 con 0x7f5c88102a60 2024-09-21T02:05:18.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.315+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78035070 con 0x7f5c88102a60 2024-09-21T02:05:18.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.316+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c88103ce0 con 0x7f5c88102a60 2024-09-21T02:05:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.319+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f5c78042030 con 0x7f5c88102a60 2024-09-21T02:05:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.320+0000 7f5c777fe640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 0x7f5c700b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.320+0000 7f5c85d9b640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 0x7f5c700b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.320+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f5c78108270 con 0x7f5c88102a60 2024-09-21T02:05:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.321+0000 7f5c85d9b640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 0x7f5c700b8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c88100d20 tx=0x7f5c7c00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.322+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5c780ce6d0 con 0x7f5c88102a60 2024-09-21T02:05:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.552+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f5c8810ab20 con 0x7f5c88102a60 2024-09-21T02:05:18.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.819+0000 7f5c777fe640 1 -- 172.21.15.16:0/1481755947 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f5c780d2ab0 con 0x7f5c88102a60 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 msgr2=0x7f5c700b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 0x7f5c700b8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c88100d20 tx=0x7f5c7c00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 msgr2=0x7f5c8819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009230 tx=0x7f5c78004270 comp rx=0 tx=0).stop 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 shutdown_connections 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5c700b6590 0x7f5c700b8aa0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c88108cf0 0x7f5c8819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c881083a0 0x7f5c8819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 --2- 172.21.15.16:0/1481755947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c88102a60 0x7f5c8819a400 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 >> 172.21.15.16:0/1481755947 conn(0x7f5c88079730 msgr2=0x7f5c88101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 shutdown_connections 2024-09-21T02:05:18.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:18.822+0000 7f5c8cc3e640 1 -- 172.21.15.16:0/1481755947 wait complete. 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:active 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T02:05:19.147 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:18 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:active 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T02:05:19.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:18 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:05:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] up:active 2024-09-21T02:05:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-21T02:05:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-21T02:05:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-21T02:05:19.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-21T02:05:19.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:18 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} : dispatch 2024-09-21T02:05:19.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs set cephfs session_timeout 300 2024-09-21T02:05:19.754 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:20.076 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-21T02:05:20.076 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:05:20.076 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:active 2024-09-21T02:05:20.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:05:20.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-21T02:05:20.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-21T02:05:20.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:19 smithi016 ceph-mon[31245]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:active 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-21T02:05:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:19 smithi161 ceph-mon[34061]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-21T02:05:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-21T02:05:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1481755947' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-21T02:05:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] up:active 2024-09-21T02:05:20.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-21T02:05:20.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-09-21T02:05:20.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-09-21T02:05:20.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:19 smithi123 ceph-mon[33453]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 -- 172.21.15.16:0/4259474829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00114460 msgr2=0x7fce00116930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 --2- 172.21.15.16:0/4259474829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00114460 0x7fce00116930 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009ca0 tx=0x7fcdf802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 -- 172.21.15.16:0/4259474829 shutdown_connections 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 --2- 172.21.15.16:0/4259474829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00114460 0x7fce00116930 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 --2- 172.21.15.16:0/4259474829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00075dd0 0x7fce000761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 --2- 172.21.15.16:0/4259474829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00077380 0x7fce000757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 -- 172.21.15.16:0/4259474829 >> 172.21.15.16:0/4259474829 conn(0x7fce000ff5e0 msgr2=0x7fce00101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 -- 172.21.15.16:0/4259474829 shutdown_connections 2024-09-21T02:05:20.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.297+0000 7fce04c75640 1 -- 172.21.15.16:0/4259474829 wait complete. 2024-09-21T02:05:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 Processor -- start 2024-09-21T02:05:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 -- start start 2024-09-21T02:05:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 0x7fce001a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 0x7fce001a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce00119f90 con 0x7fce00075dd0 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce00119e10 con 0x7fce00077380 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fce04c75640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fce0011a110 con 0x7fce00114460 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fcdfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fcdfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39706/0 (socket says 172.21.15.16:39706) 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fcdfe59c640 1 -- 172.21.15.16:0/2751706292 learned_addr learned my addr 172.21.15.16:0/2751706292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.298+0000 7fcdfed9d640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 0x7fce001a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfdd9b640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 0x7fce001a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 -- 172.21.15.16:0/2751706292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 msgr2=0x7fce001a6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 0x7fce001a6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 -- 172.21.15.16:0/2751706292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 msgr2=0x7fce001a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 0x7fce001a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 -- 172.21.15.16:0/2751706292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce001a6890 con 0x7fce00075dd0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfed9d640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 0x7fce001a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfdd9b640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 0x7fce001a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdfe59c640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000d950 tx=0x7fcdf000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0017030 con 0x7fce00075dd0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce001a6b80 con 0x7fce00075dd0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcdf0004020 con 0x7fce00075dd0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00055d0 con 0x7fce00075dd0 2024-09-21T02:05:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.299+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce001ad2e0 con 0x7fce00075dd0 2024-09-21T02:05:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.300+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce000761e0 con 0x7fce00075dd0 2024-09-21T02:05:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.302+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7fcdf0009a30 con 0x7fce00075dd0 2024-09-21T02:05:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.303+0000 7fcdef7fe640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 0x7fcde80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.303+0000 7fcdfdd9b640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 0x7fcde80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.303+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fcdf00e78b0 con 0x7fce00075dd0 2024-09-21T02:05:20.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.304+0000 7fcdfdd9b640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 0x7fcde80b8a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce00116360 tx=0x7fcdf4005e50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:20.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.306+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcdf00add10 con 0x7fce00075dd0 2024-09-21T02:05:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.534+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fce001a1a50 con 0x7fce00075dd0 2024-09-21T02:05:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.732+0000 7fcdef7fe640 1 -- 172.21.15.16:0/2751706292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7fcdf00b20f0 con 0x7fce00075dd0 2024-09-21T02:05:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 msgr2=0x7fcde80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 0x7fcde80b8a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce00116360 tx=0x7fcdf4005e50 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 msgr2=0x7fce001a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000d950 tx=0x7fcdf000de10 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 shutdown_connections 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcde80b6560 0x7fcde80b8a70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce00114460 0x7fce001a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce00077380 0x7fce001a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 --2- 172.21.15.16:0/2751706292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce00075dd0 0x7fce001a0920 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 >> 172.21.15.16:0/2751706292 conn(0x7fce000ff5e0 msgr2=0x7fce00110360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 shutdown_connections 2024-09-21T02:05:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:20.736+0000 7fce04c75640 1 -- 172.21.15.16:0/2751706292 wait complete. 2024-09-21T02:05:20.829 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:05:20.563+0000 7f4c33c91640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-21T02:05:20.829 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:05:20.563+0000 7f4c33c91640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-21T02:05:20.829 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:05:20.563+0000 7f4c33c91640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-21T02:05:21.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:active 2024-09-21T02:05:21.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: Cluster is now healthy 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:active 2024-09-21T02:05:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:20 smithi016 ceph-mon[31245]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:active 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: Cluster is now healthy 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:05:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:05:21.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:active 2024-09-21T02:05:21.166 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:20 smithi161 ceph-mon[34061]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:05:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] up:active 2024-09-21T02:05:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: Cluster is now healthy 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} : dispatch 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2751706292' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] up:active 2024-09-21T02:05:21.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:20 smithi123 ceph-mon[33453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:05:21.458 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-09-21T02:05:21.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs subvolumegroup create cephfs qa 2024-09-21T02:05:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:22.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:21 smithi016 ceph-mon[31245]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-09-21T02:05:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:21 smithi161 ceph-mon[34061]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-09-21T02:05:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:21 smithi123 ceph-mon[33453]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-09-21T02:05:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 -- 172.21.15.16:0/3557048067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 msgr2=0x7f46ec1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 --2- 172.21.15.16:0/3557048067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec1087b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009900 tx=0x7f46dc02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:22.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 -- 172.21.15.16:0/3557048067 shutdown_connections 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 --2- 172.21.15.16:0/3557048067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec108cf0 0x7f46ec10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 --2- 172.21.15.16:0/3557048067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec1087b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 --2- 172.21.15.16:0/3557048067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec102a60 0x7f46ec102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 -- 172.21.15.16:0/3557048067 >> 172.21.15.16:0/3557048067 conn(0x7f46ec079730 msgr2=0x7f46ec079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 -- 172.21.15.16:0/3557048067 shutdown_connections 2024-09-21T02:05:22.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.271+0000 7f46f3ada640 1 -- 172.21.15.16:0/3557048067 wait complete. 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.272+0000 7f46f3ada640 1 Processor -- start 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.272+0000 7f46f3ada640 1 -- start start 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.272+0000 7f46f3ada640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 0x7f46ec19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.272+0000 7f46f3ada640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f3ada640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec108cf0 0x7f46ec19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f3ada640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec113930 con 0x7f46ec1083a0 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f3ada640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1137b0 con 0x7f46ec102a60 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f3ada640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec113ab0 con 0x7f46ec108cf0 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39726/0 (socket says 172.21.15.16:39726) 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 -- 172.21.15.16:0/295165184 learned_addr learned my addr 172.21.15.16:0/295165184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f2077640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec108cf0 0x7f46ec19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1876640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 0x7f46ec19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 -- 172.21.15.16:0/295165184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec108cf0 msgr2=0x7f46ec19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec108cf0 0x7f46ec19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 -- 172.21.15.16:0/295165184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 msgr2=0x7f46ec19a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 0x7f46ec19a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 -- 172.21.15.16:0/295165184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ec1a03b0 con 0x7f46ec1083a0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1876640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 0x7f46ec19a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.273+0000 7f46f1075640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005940 tx=0x7f46dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.274+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc03f030 con 0x7f46ec1083a0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.274+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46dc003c90 con 0x7f46ec1083a0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.274+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ec1a0640 con 0x7f46ec1083a0 2024-09-21T02:05:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.274+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc035070 con 0x7f46ec1083a0 2024-09-21T02:05:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.274+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ec10fe90 con 0x7f46ec1083a0 2024-09-21T02:05:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.275+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ec103ce0 con 0x7f46ec1083a0 2024-09-21T02:05:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.276+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f46dc042030 con 0x7f46ec1083a0 2024-09-21T02:05:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.277+0000 7f46e2ffd640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 0x7f46d80b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.277+0000 7f46f1876640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 0x7f46d80b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.277+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f46dc107d00 con 0x7f46ec1083a0 2024-09-21T02:05:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.278+0000 7f46f1876640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 0x7f46d80b8a70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46e80058c0 tx=0x7f46e8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:22.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.282+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46dc0ce160 con 0x7f46ec1083a0 2024-09-21T02:05:22.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.494+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f46ec063800 con 0x7f46d80b6560 2024-09-21T02:05:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.532+0000 7f46e2ffd640 1 -- 172.21.15.16:0/295165184 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f46ec063800 con 0x7f46d80b6560 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 msgr2=0x7f46d80b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 0x7f46d80b8a70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46e80058c0 tx=0x7f46e8005850 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 msgr2=0x7f46ec19a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005940 tx=0x7f46dc0042f0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 shutdown_connections 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46d80b6560 0x7f46d80b8a70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46ec108cf0 0x7f46ec19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec1083a0 0x7f46ec19a9e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 --2- 172.21.15.16:0/295165184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46ec102a60 0x7f46ec19a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 >> 172.21.15.16:0/295165184 conn(0x7f46ec079730 msgr2=0x7f46ec104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 shutdown_connections 2024-09-21T02:05:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:22.534+0000 7f46f3ada640 1 -- 172.21.15.16:0/295165184 wait complete. 2024-09-21T02:05:22.836 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:05:22.499+0000 7f5959936640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:05:22.837 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:05:22.499+0000 7f5959936640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:05:22.837 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:05:22.499+0000 7f5959936640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:05:22.837 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:05:22.499+0000 7f5959936640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:05:22.837 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:05:22.499+0000 7f5959936640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:05:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-mon[31245]: pgmap v297: 129 pgs: 129 active+clean; 586 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-21T02:05:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:22 smithi016 ceph-mon[31245]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:23.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:22 smithi161 ceph-mon[34061]: pgmap v297: 129 pgs: 129 active+clean; 586 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-21T02:05:23.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:22 smithi161 ceph-mon[34061]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:22 smithi123 ceph-mon[33453]: pgmap v297: 129 pgs: 129 active+clean; 586 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-21T02:05:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:22 smithi123 ceph-mon[33453]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:23.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-21T02:05:23.547 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- 172.21.15.16:0/1237118537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 msgr2=0x7f4980102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 --2- 172.21.15.16:0/1237118537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f4980102ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f497c009900 tx=0x7f497c02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- 172.21.15.16:0/1237118537 shutdown_connections 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 --2- 172.21.15.16:0/1237118537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4980108d00 0x7f498010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 --2- 172.21.15.16:0/1237118537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49801083b0 0x7f49801087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 --2- 172.21.15.16:0/1237118537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f4980102ed0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- 172.21.15.16:0/1237118537 >> 172.21.15.16:0/1237118537 conn(0x7f4980079790 msgr2=0x7f4980079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- 172.21.15.16:0/1237118537 shutdown_connections 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- 172.21.15.16:0/1237118537 wait complete. 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 Processor -- start 2024-09-21T02:05:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.083+0000 7f4987a21640 1 -- start start 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801083b0 0x7f4980199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980108d00 0x7f498019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4980112870 con 0x7f4980102ac0 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49801126f0 con 0x7f4980108d00 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4987a21640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49801129f0 con 0x7f49801083b0 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39752/0 (socket says 172.21.15.16:39752) 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 -- 172.21.15.16:0/2670052276 learned_addr learned my addr 172.21.15.16:0/2670052276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4985fbe640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980108d00 0x7f498019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f4984fbc640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801083b0 0x7f4980199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 -- 172.21.15.16:0/2670052276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801083b0 msgr2=0x7f4980199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801083b0 0x7f4980199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 -- 172.21.15.16:0/2670052276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980108d00 msgr2=0x7f498019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980108d00 0x7f498019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.084+0000 7f49857bd640 1 -- 172.21.15.16:0/2670052276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498019f310 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f49857bd640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f497c009230 tx=0x7f497c0319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c046070 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f497c004350 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498019f5a0 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c035050 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.085+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498006b820 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.086+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4980103cf0 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.087+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 154529+0+0 (secure 0 0 0) 0x7f497c049070 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.088+0000 7f49767fc640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 0x7f496c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.089+0000 7f4984fbc640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 0x7f496c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.089+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f497c109550 con 0x7f4980102ac0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.089+0000 7f4984fbc640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 0x7f496c0b8af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4980079be0 tx=0x7f4970005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:24.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.092+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f497c050020 con 0x7f4980102ac0 2024-09-21T02:05:24.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.307+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f498006bb40 con 0x7f496c0b65e0 2024-09-21T02:05:24.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.336+0000 7f49767fc640 1 -- 172.21.15.16:0/2670052276 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f498006bb40 con 0x7f496c0b65e0 2024-09-21T02:05:24.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 msgr2=0x7f496c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:24.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 0x7f496c0b8af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4980079be0 tx=0x7f4970005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:24.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 msgr2=0x7f49801993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:24.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f497c009230 tx=0x7f497c0319a0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 shutdown_connections 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f496c0b65e0 0x7f496c0b8af0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4980108d00 0x7f498019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49801083b0 0x7f4980199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 --2- 172.21.15.16:0/2670052276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4980102ac0 0x7f49801993a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.338+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 >> 172.21.15.16:0/2670052276 conn(0x7f4980079790 msgr2=0x7f49801065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.339+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 shutdown_connections 2024-09-21T02:05:24.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:24.339+0000 7f4987a21640 1 -- 172.21.15.16:0/2670052276 wait complete. 2024-09-21T02:05:25.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-21T02:05:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:25.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:25 smithi016 ceph-mon[31245]: pgmap v298: 129 pgs: 129 active+clean; 594 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.9 KiB/s wr, 18 op/s 2024-09-21T02:05:25.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:25 smithi016 ceph-mon[31245]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:25.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:25 smithi161 ceph-mon[34061]: pgmap v298: 129 pgs: 129 active+clean; 594 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.9 KiB/s wr, 18 op/s 2024-09-21T02:05:25.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:25 smithi161 ceph-mon[34061]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:25.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:25 smithi123 ceph-mon[33453]: pgmap v298: 129 pgs: 129 active+clean; 594 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.9 KiB/s wr, 18 op/s 2024-09-21T02:05:25.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:25 smithi123 ceph-mon[33453]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.804+0000 7fa707a50640 1 -- 172.21.15.16:0/3661101069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7001094a0 msgr2=0x7fa7001098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.804+0000 7fa707a50640 1 --2- 172.21.15.16:0/3661101069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7001094a0 0x7fa7001098b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009900 tx=0x7fa6f002a410 comp rx=0 tx=0).stop 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 -- 172.21.15.16:0/3661101069 shutdown_connections 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 --2- 172.21.15.16:0/3661101069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700109df0 0x7fa700110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 --2- 172.21.15.16:0/3661101069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7001094a0 0x7fa7001098b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 --2- 172.21.15.16:0/3661101069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa700103b60 0x7fa700103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 -- 172.21.15.16:0/3661101069 >> 172.21.15.16:0/3661101069 conn(0x7fa70007b850 msgr2=0x7fa70007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 -- 172.21.15.16:0/3661101069 shutdown_connections 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 -- 172.21.15.16:0/3661101069 wait complete. 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 Processor -- start 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.805+0000 7fa707a50640 1 -- start start 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700103b60 0x7fa70019c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7001094a0 0x7fa70019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa700113960 con 0x7fa700109df0 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001137e0 con 0x7fa700103b60 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa707a50640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa700113ae0 con 0x7fa7001094a0 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39770/0 (socket says 172.21.15.16:39770) 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 -- 172.21.15.16:0/3737996848 learned_addr learned my addr 172.21.15.16:0/3737996848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa7057ec640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700103b60 0x7fa70019c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa704feb640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7001094a0 0x7fa70019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 -- 172.21.15.16:0/3737996848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7001094a0 msgr2=0x7fa70019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7001094a0 0x7fa70019cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 -- 172.21.15.16:0/3737996848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700103b60 msgr2=0x7fa70019c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700103b60 0x7fa70019c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 -- 172.21.15.16:0/3737996848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7001a2520 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.806+0000 7fa705fed640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800eaa0 tx=0x7fa6f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.807+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800cdb0 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.807+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6f8005020 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.807+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7001a2810 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.807+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f80116d0 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.807+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7001a8f70 con 0x7fa700109df0 2024-09-21T02:05:25.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.808+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa700103f70 con 0x7fa700109df0 2024-09-21T02:05:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.809+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fa6f80051d0 con 0x7fa700109df0 2024-09-21T02:05:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.810+0000 7fa6f67fc640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 0x7fa6ec0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.810+0000 7fa7057ec640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 0x7fa6ec0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.810+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fa6f80e7e80 con 0x7fa700109df0 2024-09-21T02:05:25.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.811+0000 7fa7057ec640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 0x7fa6ec0b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00bbe0 tx=0x7fa6fc00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:25.813+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa6f8016030 con 0x7fa700109df0 2024-09-21T02:05:26.020 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:05:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:05:25.564+0000 7f4c33c91640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-21T02:05:26.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.031+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fa7001059e0 con 0x7fa6ec0b65f0 2024-09-21T02:05:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.039+0000 7fa6f67fc640 1 -- 172.21.15.16:0/3737996848 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa7001059e0 con 0x7fa6ec0b65f0 2024-09-21T02:05:26.040 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/fcbd249e-f272-4381-beda-18705a85f32c 2024-09-21T02:05:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 msgr2=0x7fa6ec0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 0x7fa6ec0b8b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00bbe0 tx=0x7fa6fc00b390 comp rx=0 tx=0).stop 2024-09-21T02:05:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 msgr2=0x7fa7001a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800eaa0 tx=0x7fa6f800ef60 comp rx=0 tx=0).stop 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 shutdown_connections 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa6ec0b65f0 0x7fa6ec0b8b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700109df0 0x7fa7001a1df0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7001094a0 0x7fa70019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 --2- 172.21.15.16:0/3737996848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa700103b60 0x7fa70019c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 >> 172.21.15.16:0/3737996848 conn(0x7fa70007b850 msgr2=0x7fa7001052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.041+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 shutdown_connections 2024-09-21T02:05:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:26.042+0000 7fa707a50640 1 -- 172.21.15.16:0/3737996848 wait complete. 2024-09-21T02:05:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:26 smithi016 ceph-mon[31245]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-09-21T02:05:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:26 smithi161 ceph-mon[34061]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-09-21T02:05:26.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:26 smithi123 ceph-mon[33453]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-09-21T02:05:26.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-21T02:05:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:27.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:27 smithi016 ceph-mon[31245]: pgmap v299: 129 pgs: 129 active+clean; 594 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.1 KiB/s wr, 19 op/s 2024-09-21T02:05:27.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:27 smithi016 ceph-mon[31245]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:27.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:27 smithi161 ceph-mon[34061]: pgmap v299: 129 pgs: 129 active+clean; 594 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.1 KiB/s wr, 19 op/s 2024-09-21T02:05:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:27 smithi161 ceph-mon[34061]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 -- 172.21.15.16:0/4242514240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc101fe0 msgr2=0x7f3bdc1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 --2- 172.21.15.16:0/4242514240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc1023f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009900 tx=0x7f3bcc02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 -- 172.21.15.16:0/4242514240 shutdown_connections 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 --2- 172.21.15.16:0/4242514240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc109e10 0x7f3bdc110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 --2- 172.21.15.16:0/4242514240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc1094c0 0x7f3bdc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 --2- 172.21.15.16:0/4242514240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc1023f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 -- 172.21.15.16:0/4242514240 >> 172.21.15.16:0/4242514240 conn(0x7f3bdc077bc0 msgr2=0x7f3bdc078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 -- 172.21.15.16:0/4242514240 shutdown_connections 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.490+0000 7f3be315e640 1 -- 172.21.15.16:0/4242514240 wait complete. 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 Processor -- start 2024-09-21T02:05:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 -- start start 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc1094c0 0x7f3bdc19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc1139b0 con 0x7f3bdc109e10 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc113830 con 0x7f3bdc1094c0 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be315e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc113b30 con 0x7f3bdc101fe0 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be16fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be16fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34174/0 (socket says 172.21.15.16:34174) 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be16fb640 1 -- 172.21.15.16:0/4172516208 learned_addr learned my addr 172.21.15.16:0/4172516208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3be0efa640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.491+0000 7f3bdbfff640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc1094c0 0x7f3bdc19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 -- 172.21.15.16:0/4172516208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc101fe0 msgr2=0x7f3bdc19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 -- 172.21.15.16:0/4172516208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc1094c0 msgr2=0x7f3bdc19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc1094c0 0x7f3bdc19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 -- 172.21.15.16:0/4172516208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc1a04b0 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be16fb640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400b970 tx=0x7f3bd400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4019070 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3bd4014440 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bdc1a07a0 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd40075d0 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.492+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bdc1a6f00 con 0x7f3bdc109e10 2024-09-21T02:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.494+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bdc1023f0 con 0x7f3bdc109e10 2024-09-21T02:05:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.495+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3bd401c070 con 0x7f3bdc109e10 2024-09-21T02:05:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.496+0000 7f3bd9ffb640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 0x7f3bc40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.496+0000 7f3be0efa640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 0x7f3bc40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.496+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f3bd40e6e40 con 0x7f3bdc109e10 2024-09-21T02:05:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.497+0000 7f3be0efa640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 0x7f3bc40b8b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0057c0 tx=0x7f3bcc0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.501+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3bd40b14c0 con 0x7f3bdc109e10 2024-09-21T02:05:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:27 smithi123 ceph-mon[33453]: pgmap v299: 129 pgs: 129 active+clean; 594 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.1 KiB/s wr, 19 op/s 2024-09-21T02:05:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:27 smithi123 ceph-mon[33453]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.720+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3bdc063800 con 0x7f3bc40b65f0 2024-09-21T02:05:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.748+0000 7f3bd9ffb640 1 -- 172.21.15.16:0/4172516208 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3bdc063800 con 0x7f3bc40b65f0 2024-09-21T02:05:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.750+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 msgr2=0x7f3bc40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.750+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 0x7f3bc40b8b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0057c0 tx=0x7f3bcc0058d0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.750+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 msgr2=0x7f3bdc19fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.750+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400b970 tx=0x7f3bd400be30 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.750+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 shutdown_connections 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3bc40b65f0 0x7f3bc40b8b00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bdc109e10 0x7f3bdc19fd80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bdc1094c0 0x7f3bdc19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 --2- 172.21.15.16:0/4172516208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3bdc101fe0 0x7f3bdc19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 >> 172.21.15.16:0/4172516208 conn(0x7f3bdc077bc0 msgr2=0x7f3bdc107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 shutdown_connections 2024-09-21T02:05:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:27.751+0000 7f3be315e640 1 -- 172.21.15.16:0/4172516208 wait complete. 2024-09-21T02:05:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:28 smithi016 ceph-mon[31245]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:28 smithi161 ceph-mon[34061]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-21T02:05:28.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:28 smithi123 ceph-mon[33453]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:29 smithi016 ceph-mon[31245]: pgmap v300: 129 pgs: 129 active+clean; 595 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.4 KiB/s wr, 21 op/s 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- 172.21.15.16:0/1797605650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9304109e10 msgr2=0x7f9304110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 --2- 172.21.15.16:0/1797605650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9304109e10 0x7f9304110640 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009ca0 tx=0x7f92fc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- 172.21.15.16:0/1797605650 shutdown_connections 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 --2- 172.21.15.16:0/1797605650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9304109e10 0x7f9304110640 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 --2- 172.21.15.16:0/1797605650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93041094c0 0x7f93041098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 --2- 172.21.15.16:0/1797605650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304101fe0 0x7f93041023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- 172.21.15.16:0/1797605650 >> 172.21.15.16:0/1797605650 conn(0x7f9304077bc0 msgr2=0x7f9304078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:29.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- 172.21.15.16:0/1797605650 shutdown_connections 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- 172.21.15.16:0/1797605650 wait complete. 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 Processor -- start 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.258+0000 7f930b704640 1 -- start start 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9304101fe0 0x7f930419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304109e10 0x7f930419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9304113980 con 0x7f93041094c0 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9304113800 con 0x7f9304109e10 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f930b704640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9304113b00 con 0x7f9304101fe0 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34186/0 (socket says 172.21.15.16:34186) 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 -- 172.21.15.16:0/3847880346 learned_addr learned my addr 172.21.15.16:0/3847880346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9309ca1640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304109e10 0x7f930419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:29.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 -- 172.21.15.16:0/3847880346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9304101fe0 msgr2=0x7f930419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9304101fe0 0x7f930419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 -- 172.21.15.16:0/3847880346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304109e10 msgr2=0x7f930419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304109e10 0x7f930419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 -- 172.21.15.16:0/3847880346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93041a02c0 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.259+0000 7f9308c9f640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f92f400e9f0 tx=0x7f92f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.260+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400ccc0 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.260+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92f40035f0 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.260+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93041a05b0 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.260+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400ce60 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.260+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93041a6d10 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.261+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93041023f0 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.262+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f92f4004000 con 0x7f93041094c0 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.263+0000 7f92fa7fc640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 0x7f92f00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:29.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.263+0000 7f93094a0640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 0x7f92f00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:29.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.263+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f92f40e7df0 con 0x7f93041094c0 2024-09-21T02:05:29.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.264+0000 7f93094a0640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 0x7f92f00b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f93000058c0 tx=0x7f9300005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:29.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.266+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f92f40ae0f0 con 0x7f93041094c0 2024-09-21T02:05:29.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:29 smithi161 ceph-mon[34061]: pgmap v300: 129 pgs: 129 active+clean; 595 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.4 KiB/s wr, 21 op/s 2024-09-21T02:05:29.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.487+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f9304063800 con 0x7f92f00b65f0 2024-09-21T02:05:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.492+0000 7f92fa7fc640 1 -- 172.21.15.16:0/3847880346 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9304063800 con 0x7f92f00b65f0 2024-09-21T02:05:29.493 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/4f2c48a4-252b-4b9e-ba21-7820a2a1631d 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 msgr2=0x7f92f00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 0x7f92f00b8b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f93000058c0 tx=0x7f9300005850 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 msgr2=0x7f930419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f92f400e9f0 tx=0x7f92f400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 shutdown_connections 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f92f00b65f0 0x7f92f00b8b00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9304109e10 0x7f930419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93041094c0 0x7f930419a8f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 --2- 172.21.15.16:0/3847880346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9304101fe0 0x7f930419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 >> 172.21.15.16:0/3847880346 conn(0x7f9304077bc0 msgr2=0x7f93041074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 shutdown_connections 2024-09-21T02:05:29.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:29.494+0000 7f930b704640 1 -- 172.21.15.16:0/3847880346 wait complete. 2024-09-21T02:05:29.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:29 smithi123 ceph-mon[33453]: pgmap v300: 129 pgs: 129 active+clean; 595 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.4 KiB/s wr, 21 op/s 2024-09-21T02:05:30.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:05:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:30 smithi016 ceph-mon[31245]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:30.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:30 smithi161 ceph-mon[34061]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:30.458 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:30.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:30 smithi123 ceph-mon[33453]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 -- 172.21.15.16:0/1439897618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df010ddf0 msgr2=0x7f0df0110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 --2- 172.21.15.16:0/1439897618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df010ddf0 0x7f0df0110300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009ca0 tx=0x7f0de802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 -- 172.21.15.16:0/1439897618 shutdown_connections 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 --2- 172.21.15.16:0/1439897618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df010ddf0 0x7f0df0110300 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 --2- 172.21.15.16:0/1439897618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df0105270 0x7f0df010d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 --2- 172.21.15.16:0/1439897618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 0x7f0df0104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 -- 172.21.15.16:0/1439897618 >> 172.21.15.16:0/1439897618 conn(0x7f0df0079730 msgr2=0x7f0df0079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 -- 172.21.15.16:0/1439897618 shutdown_connections 2024-09-21T02:05:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.973+0000 7f0df7264640 1 -- 172.21.15.16:0/1439897618 wait complete. 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.974+0000 7f0df7264640 1 Processor -- start 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.974+0000 7f0df7264640 1 -- start start 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 0x7f0df019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df010ddf0 0x7f0df019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0df0113990 con 0x7f0df0105270 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0df0113810 con 0x7f0df0104840 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df7264640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0df0113b10 con 0x7f0df010ddf0 2024-09-21T02:05:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34200/0 (socket says 172.21.15.16:34200) 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 -- 172.21.15.16:0/2764914456 learned_addr learned my addr 172.21.15.16:0/2764914456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df5801640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df010ddf0 0x7f0df019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0df5000640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 0x7f0df019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 -- 172.21.15.16:0/2764914456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df010ddf0 msgr2=0x7f0df019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df010ddf0 0x7f0df019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 -- 172.21.15.16:0/2764914456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 msgr2=0x7f0df019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 0x7f0df019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 -- 172.21.15.16:0/2764914456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df01a0350 con 0x7f0df0105270 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0deffff640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9c0 tx=0x7f0dd800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.975+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800ccc0 con 0x7f0df0105270 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.976+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dd80054b0 con 0x7f0df0105270 2024-09-21T02:05:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.976+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df01a0640 con 0x7f0df0105270 2024-09-21T02:05:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.976+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8010430 con 0x7f0df0105270 2024-09-21T02:05:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.976+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df01a6da0 con 0x7f0df0105270 2024-09-21T02:05:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.977+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df0104c50 con 0x7f0df0105270 2024-09-21T02:05:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.978+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0dd8004f20 con 0x7f0df0105270 2024-09-21T02:05:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.979+0000 7f0dedffb640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 0x7f0ddc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.979+0000 7f0df5000640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 0x7f0ddc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.980+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f0dd80e7de0 con 0x7f0df0105270 2024-09-21T02:05:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.980+0000 7f0df5000640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 0x7f0ddc0b8b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0df006bc00 tx=0x7f0de0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:30.984+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0dd80ae080 con 0x7f0df0105270 2024-09-21T02:05:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:31 smithi016 ceph-mon[31245]: pgmap v301: 129 pgs: 129 active+clean; 631 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-09-21T02:05:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.211+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0df01a70e0 con 0x7f0df0105270 2024-09-21T02:05:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.212+0000 7f0dedffb640 1 -- 172.21.15.16:0/2764914456 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7f0dd80b2460 con 0x7f0df0105270 2024-09-21T02:05:31.213 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:31.213 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:05:31.213 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 msgr2=0x7f0ddc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 0x7f0ddc0b8b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0df006bc00 tx=0x7f0de0005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 msgr2=0x7f0df019a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9c0 tx=0x7f0dd800ee80 comp rx=0 tx=0).stop 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 shutdown_connections 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ddc0b6670 0x7f0ddc0b8b80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0df010ddf0 0x7f0df019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:31.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df0105270 0x7f0df019a980 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 --2- 172.21.15.16:0/2764914456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df0104840 0x7f0df019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 >> 172.21.15.16:0/2764914456 conn(0x7f0df0079730 msgr2=0x7f0df0107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 shutdown_connections 2024-09-21T02:05:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:31.214+0000 7f0df7264640 1 -- 172.21.15.16:0/2764914456 wait complete. 2024-09-21T02:05:31.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:31 smithi161 ceph-mon[34061]: pgmap v301: 129 pgs: 129 active+clean; 631 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-09-21T02:05:31.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:31 smithi123 ceph-mon[33453]: pgmap v301: 129 pgs: 129 active+clean; 631 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-09-21T02:05:31.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:05:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:32 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2764914456' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:05:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:32 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:05:32.406 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:32 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2764914456' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:05:32.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:32 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:05:32.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:32 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2764914456' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:05:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:32 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:05:32.768 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:33.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:33 smithi016 ceph-mon[31245]: pgmap v302: 129 pgs: 129 active+clean; 631 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 KiB/s wr, 23 op/s 2024-09-21T02:05:33.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:33 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.153 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:33 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.154 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:33 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.154 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:33 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.244+0000 7ff005468640 1 -- 172.21.15.16:0/2480044176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0001094f0 msgr2=0x7ff000109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.244+0000 7ff005468640 1 --2- 172.21.15.16:0/2480044176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0001094f0 0x7ff000109900 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009900 tx=0x7fefe802a410 comp rx=0 tx=0).stop 2024-09-21T02:05:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 -- 172.21.15.16:0/2480044176 shutdown_connections 2024-09-21T02:05:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 --2- 172.21.15.16:0/2480044176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff000109e40 0x7ff000110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 --2- 172.21.15.16:0/2480044176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0001094f0 0x7ff000109900 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 --2- 172.21.15.16:0/2480044176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 0x7ff000103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 -- 172.21.15.16:0/2480044176 >> 172.21.15.16:0/2480044176 conn(0x7ff0000ff600 msgr2=0x7ff000101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 -- 172.21.15.16:0/2480044176 shutdown_connections 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 -- 172.21.15.16:0/2480044176 wait complete. 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 Processor -- start 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.245+0000 7ff005468640 1 -- start start 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 0x7ff00019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0001094f0 0x7ff00019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0001139b0 con 0x7ff000109e40 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff000113830 con 0x7ff0001094f0 2024-09-21T02:05:33.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7ff005468640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff000113b30 con 0x7ff000103bb0 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34220/0 (socket says 172.21.15.16:34220) 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 -- 172.21.15.16:0/709347906 learned_addr learned my addr 172.21.15.16:0/709347906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7feffe7fc640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0001094f0 0x7ff00019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7feffeffd640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 0x7ff00019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 -- 172.21.15.16:0/709347906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 msgr2=0x7ff00019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 0x7ff00019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 -- 172.21.15.16:0/709347906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0001094f0 msgr2=0x7ff00019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0001094f0 0x7ff00019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 -- 172.21.15.16:0/709347906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0001a03f0 con 0x7ff000109e40 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.246+0000 7fefff7fe640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feff800eb10 tx=0x7feff800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.247+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8004b20 con 0x7ff000109e40 2024-09-21T02:05:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.247+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0001a06e0 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.247+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feff8004cc0 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.247+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8010690 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.247+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0001a6e40 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.248+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff000104e30 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.249+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7feff8005000 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.250+0000 7feff7fff640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 0x7fefec0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.250+0000 7feffeffd640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 0x7fefec0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.250+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7feff80e8090 con 0x7ff000109e40 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.251+0000 7feffeffd640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 0x7fefec0b8b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feff0009bd0 tx=0x7feff0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:33.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.253+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feff8056710 con 0x7ff000109e40 2024-09-21T02:05:33.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:33 smithi161 ceph-mon[34061]: pgmap v302: 129 pgs: 129 active+clean; 631 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 KiB/s wr, 23 op/s 2024-09-21T02:05:33.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:33 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:33 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:33 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:33 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.461+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff00002e640 con 0x7ff000109e40 2024-09-21T02:05:33.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.462+0000 7feff7fff640 1 -- 172.21.15.16:0/709347906 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25057 (secure 0 0 0) 0x7feff80ae330 con 0x7ff000109e40 2024-09-21T02:05:33.463 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:33.464 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:53.733185+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 msgr2=0x7fefec0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 0x7fefec0b8b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feff0009bd0 tx=0x7feff0005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 msgr2=0x7ff00019fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feff800eb10 tx=0x7feff800c2d0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 shutdown_connections 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fefec0b65f0 0x7fefec0b8b00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff000109e40 0x7ff00019fcc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0001094f0 0x7ff00019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 --2- 172.21.15.16:0/709347906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff000103bb0 0x7ff00019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 >> 172.21.15.16:0/709347906 conn(0x7ff0000ff600 msgr2=0x7ff000101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 shutdown_connections 2024-09-21T02:05:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:33.465+0000 7ff005468640 1 -- 172.21.15.16:0/709347906 wait complete. 2024-09-21T02:05:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:33 smithi123 ceph-mon[33453]: pgmap v302: 129 pgs: 129 active+clean; 631 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 KiB/s wr, 23 op/s 2024-09-21T02:05:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:33 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:33 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:33 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:33 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:34.151 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:05:34.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:05:34.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:34 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/709347906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:34.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:34 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/709347906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:34.432 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:34.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:34 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/709347906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/3100258136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 msgr2=0x7f9f88104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/3100258136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f88104c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009900 tx=0x7f9f7c02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/3100258136 shutdown_connections 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/3100258136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f8810de00 0x7f9f88110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/3100258136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f88105280 0x7f9f8810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/3100258136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f88104c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/3100258136 >> 172.21.15.16:0/3100258136 conn(0x7f9f88079710 msgr2=0x7f9f88079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/3100258136 shutdown_connections 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/3100258136 wait complete. 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 Processor -- start 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.939+0000 7f9f8ef7c640 1 -- start start 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f88105280 0x7f9f8819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8810de00 0x7f9f8819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88113950 con 0x7f9f88104850 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881137d0 con 0x7f9f8810de00 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8ef7c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88113ad0 con 0x7f9f88105280 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34242/0 (socket says 172.21.15.16:34242) 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 -- 172.21.15.16:0/968119745 learned_addr learned my addr 172.21.15.16:0/968119745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8d519640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8810de00 0x7f9f8819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f87fff640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f88105280 0x7f9f8819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 -- 172.21.15.16:0/968119745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f88105280 msgr2=0x7f9f8819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f88105280 0x7f9f8819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 -- 172.21.15.16:0/968119745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8810de00 msgr2=0x7f9f8819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8810de00 0x7f9f8819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.940+0000 7f9f8cd18640 1 -- 172.21.15.16:0/968119745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f881a0310 con 0x7f9f88104850 2024-09-21T02:05:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f8cd18640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009230 tx=0x7f9f7c0319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c03f030 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9f7c0043d0 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f881a05a0 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c035070 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.941+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f881a6d60 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.942+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8806c220 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.944+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f9f7c042030 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.944+0000 7f9f85ffb640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 0x7f9f780b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.944+0000 7f9f87fff640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 0x7f9f780b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.944+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f9f7c1081a0 con 0x7f9f88104850 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.945+0000 7f9f87fff640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 0x7f9f780b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102330 tx=0x7f9f74005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:34.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:34.948+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9f7c0ce440 con 0x7f9f88104850 2024-09-21T02:05:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.133+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f88063800 con 0x7f9f780b65f0 2024-09-21T02:05:35.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.140+0000 7f9f85ffb640 1 -- 172.21.15.16:0/968119745 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340636 (secure 0 0 0) 0x7f9f88063800 con 0x7f9f780b65f0 2024-09-21T02:05:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:05:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 msgr2=0x7f9f780b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 0x7f9f780b8b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102330 tx=0x7f9f74005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:35.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 msgr2=0x7f9f8819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009230 tx=0x7f9f7c0319a0 comp rx=0 tx=0).stop 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 shutdown_connections 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9f780b65f0 0x7f9f780b8b00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8810de00 0x7f9f8819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f88105280 0x7f9f8819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 --2- 172.21.15.16:0/968119745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f88104850 0x7f9f8819a3a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.143+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 >> 172.21.15.16:0/968119745 conn(0x7f9f88079710 msgr2=0x7f9f88100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.144+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 shutdown_connections 2024-09-21T02:05:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:35.144+0000 7f9f8ef7c640 1 -- 172.21.15.16:0/968119745 wait complete. 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:05:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:35 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-21T02:05:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:05:35.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:05:35.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:35 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: pgmap v303: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:05:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:35 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:05:35.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-21T02:05:34.030112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655132,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":246,"num_read_kb":223,"num_write":369,"num_write_kb":2287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":393900,"kb_used_data":11372,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547924,"statfs":{"total":1151940427776,"available":1151537074176,"internally_reserved":0,"allocated":11644928,"data_stored":7707410,"data_compressed":393510,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54826,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":151,"num_read_kb":142,"num_write":99,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003816"},"pg_stats":[{"pgid":"2.1f","version":"84'13","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:30.758194+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:05:30.758194+0000","last_peered":"2024-09-21T02:05:30.758194+0000","last_clean":"2024-09-21T02:05:30.758194+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:05:30.758194+0000","last_undegraded":"2024-09-21T02:05:30.758194+0000","last_fullsized":"2024-09-21T02:05:30.758194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140454+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:04:54.140454+0000","last_peered":"2024-09-21T02:04:54.140454+0000","last_clean":"2024-09-21T02:04:54.140454+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:04:54.140454+0000","last_undegraded":"2024-09-21T02:04:54.140454+0000","last_fullsized":"2024-09-21T02:04:54.140454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737616+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:04:53.737616+0000","last_peered":"2024-09-21T02:04:53.737616+0000","last_clean":"2024-09-21T02:04:53.737616+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:04:53.737616+0000","last_undegraded":"2024-09-21T02:04:53.737616+0000","last_fullsized":"2024-09-21T02:04:53.737616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.142014+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:04:54.142014+0000","last_peered":"2024-09-21T02:04:54.142014+0000","last_clean":"2024-09-21T02:04:54.142014+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:04:54.142014+0000","last_undegraded":"2024-09-21T02:04:54.142014+0000","last_fullsized":"2024-09-21T02:04:54.142014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745962+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:04:53.745962+0000","last_peered":"2024-09-21T02:04:53.745962+0000","last_clean":"2024-09-21T02:04:53.745962+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:04:53.745962+0000","last_undegraded":"2024-09-21T02:04:53.745962+0000","last_fullsized":"2024-09-21T02:04:53.745962+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737915+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:04:53.737915+0000","last_peered":"2024-09-21T02:04:53.737915+0000","last_clean":"2024-09-21T02:04:53.737915+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:04:53.737915+0000","last_undegraded":"2024-09-21T02:04:53.737915+0000","last_fullsized":"2024-09-21T02:04:53.737915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.741060+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:20.741060+0000","last_peered":"2024-09-21T02:05:20.741060+0000","last_clean":"2024-09-21T02:05:20.741060+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:20.741060+0000","last_undegraded":"2024-09-21T02:05:20.741060+0000","last_fullsized":"2024-09-21T02:05:20.741060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141378+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:04:54.141378+0000","last_peered":"2024-09-21T02:04:54.141378+0000","last_clean":"2024-09-21T02:04:54.141378+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:04:54.141378+0000","last_undegraded":"2024-09-21T02:04:54.141378+0000","last_fullsized":"2024-09-21T02:04:54.141378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738771+0000","last_change":"2024-09-21T02:04:40.694714+0000","last_active":"2024-09-21T02:04:53.738771+0000","last_peered":"2024-09-21T02:04:53.738771+0000","last_clean":"2024-09-21T02:04:53.738771+0000","last_became_active":"2024-09-21T02:04:40.694146+0000","last_became_peered":"2024-09-21T02:04:40.694146+0000","last_unstale":"2024-09-21T02:04:53.738771+0000","last_undegraded":"2024-09-21T02:04:53.738771+0000","last_fullsized":"2024-09-21T02:04:53.738771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:58.001324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.846610+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:19.846610+0000","last_peered":"2024-09-21T02:05:19.846610+0000","last_clean":"2024-09-21T02:05:19.846610+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:19.846610+0000","last_undegraded":"2024-09-21T02:05:19.846610+0000","last_fullsized":"2024-09-21T02:05:19.846610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.143250+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:04:54.143250+0000","last_peered":"2024-09-21T02:04:54.143250+0000","last_clean":"2024-09-21T02:04:54.143250+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:04:54.143250+0000","last_undegraded":"2024-09-21T02:04:54.143250+0000","last_fullsized":"2024-09-21T02:04:54.143250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738122+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:04:53.738122+0000","last_peered":"2024-09-21T02:04:53.738122+0000","last_clean":"2024-09-21T02:04:53.738122+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:04:53.738122+0000","last_undegraded":"2024-09-21T02:04:53.738122+0000","last_fullsized":"2024-09-21T02:04:53.738122+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742775+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:04:53.742775+0000","last_peered":"2024-09-21T02:04:53.742775+0000","last_clean":"2024-09-21T02:04:53.742775+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:04:53.742775+0000","last_undegraded":"2024-09-21T02:04:53.742775+0000","last_fullsized":"2024-09-21T02:04:53.742775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:16.818463+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:16.818463+0000","last_peered":"2024-09-21T02:05:16.818463+0000","last_clean":"2024-09-21T02:05:16.818463+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:16.818463+0000","last_undegraded":"2024-09-21T02:05:16.818463+0000","last_fullsized":"2024-09-21T02:05:16.818463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738517+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:04:53.738517+0000","last_peered":"2024-09-21T02:04:53.738517+0000","last_clean":"2024-09-21T02:04:53.738517+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:04:53.738517+0000","last_undegraded":"2024-09-21T02:04:53.738517+0000","last_fullsized":"2024-09-21T02:04:53.738517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140418+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:04:54.140418+0000","last_peered":"2024-09-21T02:04:54.140418+0000","last_clean":"2024-09-21T02:04:54.140418+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:04:54.140418+0000","last_undegraded":"2024-09-21T02:04:54.140418+0000","last_fullsized":"2024-09-21T02:04:54.140418+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837247+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:05:18.837247+0000","last_peered":"2024-09-21T02:05:18.837247+0000","last_clean":"2024-09-21T02:05:18.837247+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:05:18.837247+0000","last_undegraded":"2024-09-21T02:05:18.837247+0000","last_fullsized":"2024-09-21T02:05:18.837247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738339+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:04:53.738339+0000","last_peered":"2024-09-21T02:04:53.738339+0000","last_clean":"2024-09-21T02:04:53.738339+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:04:53.738339+0000","last_undegraded":"2024-09-21T02:04:53.738339+0000","last_fullsized":"2024-09-21T02:04:53.738339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740710+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:04:53.740710+0000","last_peered":"2024-09-21T02:04:53.740710+0000","last_clean":"2024-09-21T02:04:53.740710+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:04:53.740710+0000","last_undegraded":"2024-09-21T02:04:53.740710+0000","last_fullsized":"2024-09-21T02:04:53.740710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843140+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:19.843140+0000","last_peered":"2024-09-21T02:05:19.843140+0000","last_clean":"2024-09-21T02:05:19.843140+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:19.843140+0000","last_undegraded":"2024-09-21T02:05:19.843140+0000","last_fullsized":"2024-09-21T02:05:19.843140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737973+0000","last_change":"2024-09-21T02:04:40.697250+0000","last_active":"2024-09-21T02:04:53.737973+0000","last_peered":"2024-09-21T02:04:53.737973+0000","last_clean":"2024-09-21T02:04:53.737973+0000","last_became_active":"2024-09-21T02:04:40.696827+0000","last_became_peered":"2024-09-21T02:04:40.696827+0000","last_unstale":"2024-09-21T02:04:53.737973+0000","last_undegraded":"2024-09-21T02:04:53.737973+0000","last_fullsized":"2024-09-21T02:04:53.737973+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:37.617204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739384+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:04:53.739384+0000","last_peered":"2024-09-21T02:04:53.739384+0000","last_clean":"2024-09-21T02:04:53.739384+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:04:53.739384+0000","last_undegraded":"2024-09-21T02:04:53.739384+0000","last_fullsized":"2024-09-21T02:04:53.739384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:16.818469+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:16.818469+0000","last_peered":"2024-09-21T02:05:16.818469+0000","last_clean":"2024-09-21T02:05:16.818469+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:16.818469+0000","last_undegraded":"2024-09-21T02:05:16.818469+0000","last_fullsized":"2024-09-21T02:05:16.818469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738446+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:04:53.738446+0000","last_peered":"2024-09-21T02:04:53.738446+0000","last_clean":"2024-09-21T02:04:53.738446+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:04:53.738446+0000","last_undegraded":"2024-09-21T02:04:53.738446+0000","last_fullsized":"2024-09-21T02:04:53.738446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745495+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:04:53.745495+0000","last_peered":"2024-09-21T02:04:53.745495+0000","last_clean":"2024-09-21T02:04:53.745495+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:04:53.745495+0000","last_undegraded":"2024-09-21T02:04:53.745495+0000","last_fullsized":"2024-09-21T02:04:53.745495+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844755+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:19.844755+0000","last_peered":"2024-09-21T02:05:19.844755+0000","last_clean":"2024-09-21T02:05:19.844755+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:19.844755+0000","last_undegraded":"2024-09-21T02:05:19.844755+0000","last_fullsized":"2024-09-21T02:05:19.844755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737791+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:04:53.737791+0000","last_peered":"2024-09-21T02:04:53.737791+0000","last_clean":"2024-09-21T02:04:53.737791+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:04:53.737791+0000","last_undegraded":"2024-09-21T02:04:53.737791+0000","last_fullsized":"2024-09-21T02:04:53.737791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140817+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:04:54.140817+0000","last_peered":"2024-09-21T02:04:54.140817+0000","last_clean":"2024-09-21T02:04:54.140817+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:04:54.140817+0000","last_undegraded":"2024-09-21T02:04:54.140817+0000","last_fullsized":"2024-09-21T02:04:54.140817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'14","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:30.662284+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:05:30.662284+0000","last_peered":"2024-09-21T02:05:30.662284+0000","last_clean":"2024-09-21T02:05:30.662284+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:05:30.662284+0000","last_undegraded":"2024-09-21T02:05:30.662284+0000","last_fullsized":"2024-09-21T02:05:30.662284+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140165+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:04:54.140165+0000","last_peered":"2024-09-21T02:04:54.140165+0000","last_clean":"2024-09-21T02:04:54.140165+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:04:54.140165+0000","last_undegraded":"2024-09-21T02:04:54.140165+0000","last_fullsized":"2024-09-21T02:04:54.140165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141772+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:04:54.141772+0000","last_peered":"2024-09-21T02:04:54.141772+0000","last_clean":"2024-09-21T02:04:54.141772+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:04:54.141772+0000","last_undegraded":"2024-09-21T02:04:54.141772+0000","last_fullsized":"2024-09-21T02:04:54.141772+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845615+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:19.845615+0000","last_peered":"2024-09-21T02:05:19.845615+0000","last_clean":"2024-09-21T02:05:19.845615+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:19.845615+0000","last_undegraded":"2024-09-21T02:05:19.845615+0000","last_fullsized":"2024-09-21T02:05:19.845615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741256+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:04:53.741256+0000","last_peered":"2024-09-21T02:04:53.741256+0000","last_clean":"2024-09-21T02:04:53.741256+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:04:53.741256+0000","last_undegraded":"2024-09-21T02:04:53.741256+0000","last_fullsized":"2024-09-21T02:04:53.741256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741019+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:04:53.741019+0000","last_peered":"2024-09-21T02:04:53.741019+0000","last_clean":"2024-09-21T02:04:53.741019+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:04:53.741019+0000","last_undegraded":"2024-09-21T02:04:53.741019+0000","last_fullsized":"2024-09-21T02:04:53.741019+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737883+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:04:53.737883+0000","last_peered":"2024-09-21T02:04:53.737883+0000","last_clean":"2024-09-21T02:04:53.737883+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:04:53.737883+0000","last_undegraded":"2024-09-21T02:04:53.737883+0000","last_fullsized":"2024-09-21T02:04:53.737883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739717+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:04:53.739717+0000","last_peered":"2024-09-21T02:04:53.739717+0000","last_clean":"2024-09-21T02:04:53.739717+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:04:53.739717+0000","last_undegraded":"2024-09-21T02:04:53.739717+0000","last_fullsized":"2024-09-21T02:04:53.739717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740866+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:04:53.740866+0000","last_peered":"2024-09-21T02:04:53.740866+0000","last_clean":"2024-09-21T02:04:53.740866+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:04:53.740866+0000","last_undegraded":"2024-09-21T02:04:53.740866+0000","last_fullsized":"2024-09-21T02:04:53.740866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":54,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:29.757511+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:05:29.757511+0000","last_peered":"2024-09-21T02:05:29.757511+0000","last_clean":"2024-09-21T02:05:29.757511+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:05:29.757511+0000","last_undegraded":"2024-09-21T02:05:29.757511+0000","last_fullsized":"2024-09-21T02:05:29.757511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141410+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:04:54.141410+0000","last_peered":"2024-09-21T02:04:54.141410+0000","last_clean":"2024-09-21T02:04:54.141410+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:04:54.141410+0000","last_undegraded":"2024-09-21T02:04:54.141410+0000","last_fullsized":"2024-09-21T02:04:54.141410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744493+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:04:53.744493+0000","last_peered":"2024-09-21T02:04:53.744493+0000","last_clean":"2024-09-21T02:04:53.744493+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:04:53.744493+0000","last_undegraded":"2024-09-21T02:04:53.744493+0000","last_fullsized":"2024-09-21T02:04:53.744493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":52,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:29.759318+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:05:29.759318+0000","last_peered":"2024-09-21T02:05:29.759318+0000","last_clean":"2024-09-21T02:05:29.759318+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:05:29.759318+0000","last_undegraded":"2024-09-21T02:05:29.759318+0000","last_fullsized":"2024-09-21T02:05:29.759318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141685+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:04:54.141685+0000","last_peered":"2024-09-21T02:04:54.141685+0000","last_clean":"2024-09-21T02:04:54.141685+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:04:54.141685+0000","last_undegraded":"2024-09-21T02:04:54.141685+0000","last_fullsized":"2024-09-21T02:04:54.141685+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141256+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:04:54.141256+0000","last_peered":"2024-09-21T02:04:54.141256+0000","last_clean":"2024-09-21T02:04:54.141256+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:04:54.141256+0000","last_undegraded":"2024-09-21T02:04:54.141256+0000","last_fullsized":"2024-09-21T02:04:54.141256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:17.826289+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:17.826289+0000","last_peered":"2024-09-21T02:05:17.826289+0000","last_clean":"2024-09-21T02:05:17.826289+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:17.826289+0000","last_undegraded":"2024-09-21T02:05:17.826289+0000","last_fullsized":"2024-09-21T02:05:17.826289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745568+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:04:53.745568+0000","last_peered":"2024-09-21T02:04:53.745568+0000","last_clean":"2024-09-21T02:04:53.745568+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:04:53.745568+0000","last_undegraded":"2024-09-21T02:04:53.745568+0000","last_fullsized":"2024-09-21T02:04:53.745568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140709+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:04:54.140709+0000","last_peered":"2024-09-21T02:04:54.140709+0000","last_clean":"2024-09-21T02:04:54.140709+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:04:54.140709+0000","last_undegraded":"2024-09-21T02:04:54.140709+0000","last_fullsized":"2024-09-21T02:04:54.140709+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.740461+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:20.740461+0000","last_peered":"2024-09-21T02:05:20.740461+0000","last_clean":"2024-09-21T02:05:20.740461+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:20.740461+0000","last_undegraded":"2024-09-21T02:05:20.740461+0000","last_fullsized":"2024-09-21T02:05:20.740461+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":106,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743276+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:04:53.743276+0000","last_peered":"2024-09-21T02:04:53.743276+0000","last_clean":"2024-09-21T02:04:53.743276+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:04:53.743276+0000","last_undegraded":"2024-09-21T02:04:53.743276+0000","last_fullsized":"2024-09-21T02:04:53.743276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743350+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:04:53.743350+0000","last_peered":"2024-09-21T02:04:53.743350+0000","last_clean":"2024-09-21T02:04:53.743350+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:04:53.743350+0000","last_undegraded":"2024-09-21T02:04:53.743350+0000","last_fullsized":"2024-09-21T02:04:53.743350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743486+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:04:53.743486+0000","last_peered":"2024-09-21T02:04:53.743486+0000","last_clean":"2024-09-21T02:04:53.743486+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:04:53.743486+0000","last_undegraded":"2024-09-21T02:04:53.743486+0000","last_fullsized":"2024-09-21T02:04:53.743486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"84'6","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:29.758348+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:05:29.758348+0000","last_peered":"2024-09-21T02:05:29.758348+0000","last_clean":"2024-09-21T02:05:29.758348+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:05:29.758348+0000","last_undegraded":"2024-09-21T02:05:29.758348+0000","last_fullsized":"2024-09-21T02:05:29.758348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140997+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:04:54.140997+0000","last_peered":"2024-09-21T02:04:54.140997+0000","last_clean":"2024-09-21T02:04:54.140997+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:04:54.140997+0000","last_undegraded":"2024-09-21T02:04:54.140997+0000","last_fullsized":"2024-09-21T02:04:54.140997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.749586+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:04:53.749586+0000","last_peered":"2024-09-21T02:04:53.749586+0000","last_clean":"2024-09-21T02:04:53.749586+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:04:53.749586+0000","last_undegraded":"2024-09-21T02:04:53.749586+0000","last_fullsized":"2024-09-21T02:04:53.749586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844266+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:05:19.844266+0000","last_peered":"2024-09-21T02:05:19.844266+0000","last_clean":"2024-09-21T02:05:19.844266+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:05:19.844266+0000","last_undegraded":"2024-09-21T02:05:19.844266+0000","last_fullsized":"2024-09-21T02:05:19.844266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745226+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:04:53.745226+0000","last_peered":"2024-09-21T02:04:53.745226+0000","last_clean":"2024-09-21T02:04:53.745226+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:04:53.745226+0000","last_undegraded":"2024-09-21T02:04:53.745226+0000","last_fullsized":"2024-09-21T02:04:53.745226+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743449+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:04:53.743449+0000","last_peered":"2024-09-21T02:04:53.743449+0000","last_clean":"2024-09-21T02:04:53.743449+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:04:53.743449+0000","last_undegraded":"2024-09-21T02:04:53.743449+0000","last_fullsized":"2024-09-21T02:04:53.743449+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843840+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:19.843840+0000","last_peered":"2024-09-21T02:05:19.843840+0000","last_clean":"2024-09-21T02:05:19.843840+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:19.843840+0000","last_undegraded":"2024-09-21T02:05:19.843840+0000","last_fullsized":"2024-09-21T02:05:19.843840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744290+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:04:53.744290+0000","last_peered":"2024-09-21T02:04:53.744290+0000","last_clean":"2024-09-21T02:04:53.744290+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:04:53.744290+0000","last_undegraded":"2024-09-21T02:04:53.744290+0000","last_fullsized":"2024-09-21T02:04:53.744290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741744+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:04:53.741744+0000","last_peered":"2024-09-21T02:04:53.741744+0000","last_clean":"2024-09-21T02:04:53.741744+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:04:53.741744+0000","last_undegraded":"2024-09-21T02:04:53.741744+0000","last_fullsized":"2024-09-21T02:04:53.741744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.820677+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:05:19.820677+0000","last_peered":"2024-09-21T02:05:19.820677+0000","last_clean":"2024-09-21T02:05:19.820677+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:05:19.820677+0000","last_undegraded":"2024-09-21T02:05:19.820677+0000","last_fullsized":"2024-09-21T02:05:19.820677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745149+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:04:53.745149+0000","last_peered":"2024-09-21T02:04:53.745149+0000","last_clean":"2024-09-21T02:04:53.745149+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:04:53.745149+0000","last_undegraded":"2024-09-21T02:04:53.745149+0000","last_fullsized":"2024-09-21T02:04:53.745149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.143763+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:04:54.143763+0000","last_peered":"2024-09-21T02:04:54.143763+0000","last_clean":"2024-09-21T02:04:54.143763+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:04:54.143763+0000","last_undegraded":"2024-09-21T02:04:54.143763+0000","last_fullsized":"2024-09-21T02:04:54.143763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845181+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:19.845181+0000","last_peered":"2024-09-21T02:05:19.845181+0000","last_clean":"2024-09-21T02:05:19.845181+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:19.845181+0000","last_undegraded":"2024-09-21T02:05:19.845181+0000","last_fullsized":"2024-09-21T02:05:19.845181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141822+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:04:54.141822+0000","last_peered":"2024-09-21T02:04:54.141822+0000","last_clean":"2024-09-21T02:04:54.141822+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:04:54.141822+0000","last_undegraded":"2024-09-21T02:04:54.141822+0000","last_fullsized":"2024-09-21T02:04:54.141822+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742465+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:04:53.742465+0000","last_peered":"2024-09-21T02:04:53.742465+0000","last_clean":"2024-09-21T02:04:53.742465+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:04:53.742465+0000","last_undegraded":"2024-09-21T02:04:53.742465+0000","last_fullsized":"2024-09-21T02:04:53.742465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:31.763600+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:05:31.763600+0000","last_peered":"2024-09-21T02:05:31.763600+0000","last_clean":"2024-09-21T02:05:31.763600+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:05:31.763600+0000","last_undegraded":"2024-09-21T02:05:31.763600+0000","last_fullsized":"2024-09-21T02:05:31.763600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738022+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:04:53.738022+0000","last_peered":"2024-09-21T02:04:53.738022+0000","last_clean":"2024-09-21T02:04:53.738022+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:04:53.738022+0000","last_undegraded":"2024-09-21T02:04:53.738022+0000","last_fullsized":"2024-09-21T02:04:53.738022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742906+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:04:53.742906+0000","last_peered":"2024-09-21T02:04:53.742906+0000","last_clean":"2024-09-21T02:04:53.742906+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:04:53.742906+0000","last_undegraded":"2024-09-21T02:04:53.742906+0000","last_fullsized":"2024-09-21T02:04:53.742906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844769+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:19.844769+0000","last_peered":"2024-09-21T02:05:19.844769+0000","last_clean":"2024-09-21T02:05:19.844769+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:19.844769+0000","last_undegraded":"2024-09-21T02:05:19.844769+0000","last_fullsized":"2024-09-21T02:05:19.844769+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741306+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:04:53.741306+0000","last_peered":"2024-09-21T02:04:53.741306+0000","last_clean":"2024-09-21T02:04:53.741306+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:04:53.741306+0000","last_undegraded":"2024-09-21T02:04:53.741306+0000","last_fullsized":"2024-09-21T02:04:53.741306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141420+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:04:54.141420+0000","last_peered":"2024-09-21T02:04:54.141420+0000","last_clean":"2024-09-21T02:04:54.141420+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:04:54.141420+0000","last_undegraded":"2024-09-21T02:04:54.141420+0000","last_fullsized":"2024-09-21T02:04:54.141420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.741159+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:20.741159+0000","last_peered":"2024-09-21T02:05:20.741159+0000","last_clean":"2024-09-21T02:05:20.741159+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:20.741159+0000","last_undegraded":"2024-09-21T02:05:20.741159+0000","last_fullsized":"2024-09-21T02:05:20.741159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744934+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:04:53.744934+0000","last_peered":"2024-09-21T02:04:53.744934+0000","last_clean":"2024-09-21T02:04:53.744934+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:04:53.744934+0000","last_undegraded":"2024-09-21T02:04:53.744934+0000","last_fullsized":"2024-09-21T02:04:53.744934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739572+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:04:53.739572+0000","last_peered":"2024-09-21T02:04:53.739572+0000","last_clean":"2024-09-21T02:04:53.739572+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:04:53.739572+0000","last_undegraded":"2024-09-21T02:04:53.739572+0000","last_fullsized":"2024-09-21T02:04:53.739572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'31","reported_seq":99,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.769477+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:05:32.769477+0000","last_peered":"2024-09-21T02:05:32.769477+0000","last_clean":"2024-09-21T02:05:32.769477+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:05:32.769477+0000","last_undegraded":"2024-09-21T02:05:32.769477+0000","last_fullsized":"2024-09-21T02:05:32.769477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56544,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":71,"num_write":33,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737182+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:04:53.737182+0000","last_peered":"2024-09-21T02:04:53.737182+0000","last_clean":"2024-09-21T02:04:53.737182+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:04:53.737182+0000","last_undegraded":"2024-09-21T02:04:53.737182+0000","last_fullsized":"2024-09-21T02:04:53.737182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141299+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:04:54.141299+0000","last_peered":"2024-09-21T02:04:54.141299+0000","last_clean":"2024-09-21T02:04:54.141299+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:04:54.141299+0000","last_undegraded":"2024-09-21T02:04:54.141299+0000","last_fullsized":"2024-09-21T02:04:54.141299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837715+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:18.837715+0000","last_peered":"2024-09-21T02:05:18.837715+0000","last_clean":"2024-09-21T02:05:18.837715+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:18.837715+0000","last_undegraded":"2024-09-21T02:05:18.837715+0000","last_fullsized":"2024-09-21T02:05:18.837715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738451+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:04:53.738451+0000","last_peered":"2024-09-21T02:04:53.738451+0000","last_clean":"2024-09-21T02:04:53.738451+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:04:53.738451+0000","last_undegraded":"2024-09-21T02:04:53.738451+0000","last_fullsized":"2024-09-21T02:04:53.738451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.746035+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:04:53.746035+0000","last_peered":"2024-09-21T02:04:53.746035+0000","last_clean":"2024-09-21T02:04:53.746035+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:04:53.746035+0000","last_undegraded":"2024-09-21T02:04:53.746035+0000","last_fullsized":"2024-09-21T02:04:53.746035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845342+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:19.845342+0000","last_peered":"2024-09-21T02:05:19.845342+0000","last_clean":"2024-09-21T02:05:19.845342+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:19.845342+0000","last_undegraded":"2024-09-21T02:05:19.845342+0000","last_fullsized":"2024-09-21T02:05:19.845342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.749629+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:04:53.749629+0000","last_peered":"2024-09-21T02:04:53.749629+0000","last_clean":"2024-09-21T02:04:53.749629+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:04:53.749629+0000","last_undegraded":"2024-09-21T02:04:53.749629+0000","last_fullsized":"2024-09-21T02:04:53.749629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141370+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:04:54.141370+0000","last_peered":"2024-09-21T02:04:54.141370+0000","last_clean":"2024-09-21T02:04:54.141370+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:04:54.141370+0000","last_undegraded":"2024-09-21T02:04:54.141370+0000","last_fullsized":"2024-09-21T02:04:54.141370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.761998+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:05:32.761998+0000","last_peered":"2024-09-21T02:05:32.761998+0000","last_clean":"2024-09-21T02:05:32.761998+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:05:32.761998+0000","last_undegraded":"2024-09-21T02:05:32.761998+0000","last_fullsized":"2024-09-21T02:05:32.761998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140370+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:04:54.140370+0000","last_peered":"2024-09-21T02:04:54.140370+0000","last_clean":"2024-09-21T02:04:54.140370+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:04:54.140370+0000","last_undegraded":"2024-09-21T02:04:54.140370+0000","last_fullsized":"2024-09-21T02:04:54.140370+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745297+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:04:53.745297+0000","last_peered":"2024-09-21T02:04:53.745297+0000","last_clean":"2024-09-21T02:04:53.745297+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:04:53.745297+0000","last_undegraded":"2024-09-21T02:04:53.745297+0000","last_fullsized":"2024-09-21T02:04:53.745297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737177+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:04:53.737177+0000","last_peered":"2024-09-21T02:04:53.737177+0000","last_clean":"2024-09-21T02:04:53.737177+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:04:53.737177+0000","last_undegraded":"2024-09-21T02:04:53.737177+0000","last_fullsized":"2024-09-21T02:04:53.737177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843775+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:19.843775+0000","last_peered":"2024-09-21T02:05:19.843775+0000","last_clean":"2024-09-21T02:05:19.843775+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:19.843775+0000","last_undegraded":"2024-09-21T02:05:19.843775+0000","last_fullsized":"2024-09-21T02:05:19.843775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741573+0000","last_change":"2024-09-21T02:04:40.695415+0000","last_active":"2024-09-21T02:04:53.741573+0000","last_peered":"2024-09-21T02:04:53.741573+0000","last_clean":"2024-09-21T02:04:53.741573+0000","last_became_active":"2024-09-21T02:04:40.694905+0000","last_became_peered":"2024-09-21T02:04:40.694905+0000","last_unstale":"2024-09-21T02:04:53.741573+0000","last_undegraded":"2024-09-21T02:04:53.741573+0000","last_fullsized":"2024-09-21T02:04:53.741573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:40.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140494+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:04:54.140494+0000","last_peered":"2024-09-21T02:04:54.140494+0000","last_clean":"2024-09-21T02:04:54.140494+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:04:54.140494+0000","last_undegraded":"2024-09-21T02:04:54.140494+0000","last_fullsized":"2024-09-21T02:04:54.140494+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:17.825059+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:17.825059+0000","last_peered":"2024-09-21T02:05:17.825059+0000","last_clean":"2024-09-21T02:05:17.825059+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:17.825059+0000","last_undegraded":"2024-09-21T02:05:17.825059+0000","last_fullsized":"2024-09-21T02:05:17.825059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837433+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:18.837433+0000","last_peered":"2024-09-21T02:05:18.837433+0000","last_clean":"2024-09-21T02:05:18.837433+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:18.837433+0000","last_undegraded":"2024-09-21T02:05:18.837433+0000","last_fullsized":"2024-09-21T02:05:18.837433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.142556+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:04:54.142556+0000","last_peered":"2024-09-21T02:04:54.142556+0000","last_clean":"2024-09-21T02:04:54.142556+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:04:54.142556+0000","last_undegraded":"2024-09-21T02:04:54.142556+0000","last_fullsized":"2024-09-21T02:04:54.142556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738807+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:04:53.738807+0000","last_peered":"2024-09-21T02:04:53.738807+0000","last_clean":"2024-09-21T02:04:53.738807+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:04:53.738807+0000","last_undegraded":"2024-09-21T02:04:53.738807+0000","last_fullsized":"2024-09-21T02:04:53.738807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'6","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.762793+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:05:32.762793+0000","last_peered":"2024-09-21T02:05:32.762793+0000","last_clean":"2024-09-21T02:05:32.762793+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:05:32.762793+0000","last_undegraded":"2024-09-21T02:05:32.762793+0000","last_fullsized":"2024-09-21T02:05:32.762793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739925+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:04:53.739925+0000","last_peered":"2024-09-21T02:04:53.739925+0000","last_clean":"2024-09-21T02:04:53.739925+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:04:53.739925+0000","last_undegraded":"2024-09-21T02:04:53.739925+0000","last_fullsized":"2024-09-21T02:04:53.739925+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738584+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:04:53.738584+0000","last_peered":"2024-09-21T02:04:53.738584+0000","last_clean":"2024-09-21T02:04:53.738584+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:04:53.738584+0000","last_undegraded":"2024-09-21T02:04:53.738584+0000","last_fullsized":"2024-09-21T02:04:53.738584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740306+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:04:53.740306+0000","last_peered":"2024-09-21T02:04:53.740306+0000","last_clean":"2024-09-21T02:04:53.740306+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:04:53.740306+0000","last_undegraded":"2024-09-21T02:04:53.740306+0000","last_fullsized":"2024-09-21T02:04:53.740306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738227+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:04:53.738227+0000","last_peered":"2024-09-21T02:04:53.738227+0000","last_clean":"2024-09-21T02:04:53.738227+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:04:53.738227+0000","last_undegraded":"2024-09-21T02:04:53.738227+0000","last_fullsized":"2024-09-21T02:04:53.738227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738237+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:04:53.738237+0000","last_peered":"2024-09-21T02:04:53.738237+0000","last_clean":"2024-09-21T02:04:53.738237+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:04:53.738237+0000","last_undegraded":"2024-09-21T02:04:53.738237+0000","last_fullsized":"2024-09-21T02:04:53.738237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745630+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:04:53.745630+0000","last_peered":"2024-09-21T02:04:53.745630+0000","last_clean":"2024-09-21T02:04:53.745630+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:04:53.745630+0000","last_undegraded":"2024-09-21T02:04:53.745630+0000","last_fullsized":"2024-09-21T02:04:53.745630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141656+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:04:54.141656+0000","last_peered":"2024-09-21T02:04:54.141656+0000","last_clean":"2024-09-21T02:04:54.141656+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:04:54.141656+0000","last_undegraded":"2024-09-21T02:04:54.141656+0000","last_fullsized":"2024-09-21T02:04:54.141656+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742150+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:04:53.742150+0000","last_peered":"2024-09-21T02:04:53.742150+0000","last_clean":"2024-09-21T02:04:53.742150+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:04:53.742150+0000","last_undegraded":"2024-09-21T02:04:53.742150+0000","last_fullsized":"2024-09-21T02:04:53.742150+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738850+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:04:53.738850+0000","last_peered":"2024-09-21T02:04:53.738850+0000","last_clean":"2024-09-21T02:04:53.738850+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:04:53.738850+0000","last_undegraded":"2024-09-21T02:04:53.738850+0000","last_fullsized":"2024-09-21T02:04:53.738850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140692+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:04:54.140692+0000","last_peered":"2024-09-21T02:04:54.140692+0000","last_clean":"2024-09-21T02:04:54.140692+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:04:54.140692+0000","last_undegraded":"2024-09-21T02:04:54.140692+0000","last_fullsized":"2024-09-21T02:04:54.140692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744457+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:04:53.744457+0000","last_peered":"2024-09-21T02:04:53.744457+0000","last_clean":"2024-09-21T02:04:53.744457+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:04:53.744457+0000","last_undegraded":"2024-09-21T02:04:53.744457+0000","last_fullsized":"2024-09-21T02:04:53.744457+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140564+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:04:54.140564+0000","last_peered":"2024-09-21T02:04:54.140564+0000","last_clean":"2024-09-21T02:04:54.140564+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:04:54.140564+0000","last_undegraded":"2024-09-21T02:04:54.140564+0000","last_fullsized":"2024-09-21T02:04:54.140564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141643+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:04:54.141643+0000","last_peered":"2024-09-21T02:04:54.141643+0000","last_clean":"2024-09-21T02:04:54.141643+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:04:54.141643+0000","last_undegraded":"2024-09-21T02:04:54.141643+0000","last_fullsized":"2024-09-21T02:04:54.141643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141681+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:04:54.141681+0000","last_peered":"2024-09-21T02:04:54.141681+0000","last_clean":"2024-09-21T02:04:54.141681+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:04:54.141681+0000","last_undegraded":"2024-09-21T02:04:54.141681+0000","last_fullsized":"2024-09-21T02:04:54.141681+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741405+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:04:53.741405+0000","last_peered":"2024-09-21T02:04:53.741405+0000","last_clean":"2024-09-21T02:04:53.741405+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:04:53.741405+0000","last_undegraded":"2024-09-21T02:04:53.741405+0000","last_fullsized":"2024-09-21T02:04:53.741405+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739214+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:04:53.739214+0000","last_peered":"2024-09-21T02:04:53.739214+0000","last_clean":"2024-09-21T02:04:53.739214+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:04:53.739214+0000","last_undegraded":"2024-09-21T02:04:53.739214+0000","last_fullsized":"2024-09-21T02:04:53.739214+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737539+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:04:53.737539+0000","last_peered":"2024-09-21T02:04:53.737539+0000","last_clean":"2024-09-21T02:04:53.737539+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:04:53.737539+0000","last_undegraded":"2024-09-21T02:04:53.737539+0000","last_fullsized":"2024-09-21T02:04:53.737539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742143+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:04:53.742143+0000","last_peered":"2024-09-21T02:04:53.742143+0000","last_clean":"2024-09-21T02:04:53.742143+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:04:53.742143+0000","last_undegraded":"2024-09-21T02:04:53.742143+0000","last_fullsized":"2024-09-21T02:04:53.742143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141821+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:04:54.141821+0000","last_peered":"2024-09-21T02:04:54.141821+0000","last_clean":"2024-09-21T02:04:54.141821+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:04:54.141821+0000","last_undegraded":"2024-09-21T02:04:54.141821+0000","last_fullsized":"2024-09-21T02:04:54.141821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745028+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:04:53.745028+0000","last_peered":"2024-09-21T02:04:53.745028+0000","last_clean":"2024-09-21T02:04:53.745028+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:04:53.745028+0000","last_undegraded":"2024-09-21T02:04:53.745028+0000","last_fullsized":"2024-09-21T02:04:53.745028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745118+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:04:53.745118+0000","last_peered":"2024-09-21T02:04:53.745118+0000","last_clean":"2024-09-21T02:04:53.745118+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:04:53.745118+0000","last_undegraded":"2024-09-21T02:04:53.745118+0000","last_fullsized":"2024-09-21T02:04:53.745118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141534+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:04:54.141534+0000","last_peered":"2024-09-21T02:04:54.141534+0000","last_clean":"2024-09-21T02:04:54.141534+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:04:54.141534+0000","last_undegraded":"2024-09-21T02:04:54.141534+0000","last_fullsized":"2024-09-21T02:04:54.141534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738569+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:04:53.738569+0000","last_peered":"2024-09-21T02:04:53.738569+0000","last_clean":"2024-09-21T02:04:53.738569+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:04:53.738569+0000","last_undegraded":"2024-09-21T02:04:53.738569+0000","last_fullsized":"2024-09-21T02:04:53.738569+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743491+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:04:53.743491+0000","last_peered":"2024-09-21T02:04:53.743491+0000","last_clean":"2024-09-21T02:04:53.743491+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:04:53.743491+0000","last_undegraded":"2024-09-21T02:04:53.743491+0000","last_fullsized":"2024-09-21T02:04:53.743491+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.748540+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:04:53.748540+0000","last_peered":"2024-09-21T02:04:53.748540+0000","last_clean":"2024-09-21T02:04:53.748540+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:04:53.748540+0000","last_undegraded":"2024-09-21T02:04:53.748540+0000","last_fullsized":"2024-09-21T02:04:53.748540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141785+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:04:54.141785+0000","last_peered":"2024-09-21T02:04:54.141785+0000","last_clean":"2024-09-21T02:04:54.141785+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:04:54.141785+0000","last_undegraded":"2024-09-21T02:04:54.141785+0000","last_fullsized":"2024-09-21T02:04:54.141785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140368+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:04:54.140368+0000","last_peered":"2024-09-21T02:04:54.140368+0000","last_clean":"2024-09-21T02:04:54.140368+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:04:54.140368+0000","last_undegraded":"2024-09-21T02:04:54.140368+0000","last_fullsized":"2024-09-21T02:04:54.140368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742612+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:04:53.742612+0000","last_peered":"2024-09-21T02:04:53.742612+0000","last_clean":"2024-09-21T02:04:53.742612+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:04:53.742612+0000","last_undegraded":"2024-09-21T02:04:53.742612+0000","last_fullsized":"2024-09-21T02:04:53.742612+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740817+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:04:53.740817+0000","last_peered":"2024-09-21T02:04:53.740817+0000","last_clean":"2024-09-21T02:04:53.740817+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:04:53.740817+0000","last_undegraded":"2024-09-21T02:04:53.740817+0000","last_fullsized":"2024-09-21T02:04:53.740817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743244+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:04:53.743244+0000","last_peered":"2024-09-21T02:04:53.743244+0000","last_clean":"2024-09-21T02:04:53.743244+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:04:53.743244+0000","last_undegraded":"2024-09-21T02:04:53.743244+0000","last_fullsized":"2024-09-21T02:04:53.743244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738442+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:04:53.738442+0000","last_peered":"2024-09-21T02:04:53.738442+0000","last_clean":"2024-09-21T02:04:53.738442+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:04:53.738442+0000","last_undegraded":"2024-09-21T02:04:53.738442+0000","last_fullsized":"2024-09-21T02:04:53.738442+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745107+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:04:53.745107+0000","last_peered":"2024-09-21T02:04:53.745107+0000","last_clean":"2024-09-21T02:04:53.745107+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:04:53.745107+0000","last_undegraded":"2024-09-21T02:04:53.745107+0000","last_fullsized":"2024-09-21T02:04:53.745107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745391+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:04:53.745391+0000","last_peered":"2024-09-21T02:04:53.745391+0000","last_clean":"2024-09-21T02:04:53.745391+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:04:53.745391+0000","last_undegraded":"2024-09-21T02:04:53.745391+0000","last_fullsized":"2024-09-21T02:04:53.745391+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742011+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:04:53.742011+0000","last_peered":"2024-09-21T02:04:53.742011+0000","last_clean":"2024-09-21T02:04:53.742011+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:04:53.742011+0000","last_undegraded":"2024-09-21T02:04:53.742011+0000","last_fullsized":"2024-09-21T02:04:53.742011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64530,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":165,"num_read_kb":154,"num_write":244,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":193590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710745,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":933888,"data_stored":481620,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1124868,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037795,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":485143,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":481313,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397548,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":1078704,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561074,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":925696,"data_stored":476942,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":1073504,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":481016,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117058,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":476259,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378695,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":535015,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542220,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":534628,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":478398,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":59186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:05:35.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:05:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:36.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:36 smithi016 ceph-mon[31245]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:36.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:36 smithi123 ceph-mon[33453]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:36.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 -- 172.21.15.16:0/1278098245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 msgr2=0x7fd77010f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:36.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 --2- 172.21.15.16:0/1278098245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd77010f530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd768009cc0 tx=0x7fd76802cad0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 -- 172.21.15.16:0/1278098245 shutdown_connections 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 --2- 172.21.15.16:0/1278098245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd77010f530 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 --2- 172.21.15.16:0/1278098245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd7701087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 --2- 172.21.15.16:0/1278098245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 0x7fd770102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 -- 172.21.15.16:0/1278098245 >> 172.21.15.16:0/1278098245 conn(0x7fd770079790 msgr2=0x7fd770079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 -- 172.21.15.16:0/1278098245 shutdown_connections 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.606+0000 7fd778570640 1 -- 172.21.15.16:0/1278098245 wait complete. 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 Processor -- start 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 -- start start 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 0x7fd77019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd77019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:36.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd770111770 con 0x7fd770108d00 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7701115f0 con 0x7fd770102ac0 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd778570640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7701118f0 con 0x7fd7701083b0 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd776b0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd776b0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34248/0 (socket says 172.21.15.16:34248) 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd776b0d640 1 -- 172.21.15.16:0/1356473091 learned_addr learned my addr 172.21.15.16:0/1356473091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.607+0000 7fd775b0b640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd77019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd77630c640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 0x7fd77019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 -- 172.21.15.16:0/1356473091 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 msgr2=0x7fd77019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd77019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 -- 172.21.15.16:0/1356473091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 msgr2=0x7fd77019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 0x7fd77019c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 -- 172.21.15.16:0/1356473091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7701a2440 con 0x7fd770108d00 2024-09-21T02:05:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd775b0b640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd77019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd776b0d640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd76802d0c0 tx=0x7fd768038bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768045070 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7701a26d0 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76802e630 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768040690 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.608+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7701a2bb0 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.609+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd770102ed0 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.611+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd76802eae0 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.612+0000 7fd7677fe640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 0x7fd75c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.612+0000 7fd77630c640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 0x7fd75c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.612+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fd7681086e0 con 0x7fd770108d00 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.613+0000 7fd77630c640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 0x7fd75c0b8b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd76c005fd0 tx=0x7fd76c005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:36.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.616+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd768034030 con 0x7fd770108d00 2024-09-21T02:05:36.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:36 smithi161 ceph-mon[34061]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.806+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd770104950 con 0x7fd75c0b6620 2024-09-21T02:05:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.812+0000 7fd7677fe640 1 -- 172.21.15.16:0/1356473091 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340632 (secure 0 0 0) 0x7fd770104950 con 0x7fd75c0b6620 2024-09-21T02:05:36.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:36.818 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 msgr2=0x7fd75c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 0x7fd75c0b8b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd76c005fd0 tx=0x7fd76c005850 comp rx=0 tx=0).stop 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 msgr2=0x7fd7701a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd76802d0c0 tx=0x7fd768038bd0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 shutdown_connections 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd75c0b6620 0x7fd75c0b8b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd770108d00 0x7fd7701a1dd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7701083b0 0x7fd77019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 --2- 172.21.15.16:0/1356473091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd770102ac0 0x7fd77019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 >> 172.21.15.16:0/1356473091 conn(0x7fd770079790 msgr2=0x7fd770104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 shutdown_connections 2024-09-21T02:05:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:36.815+0000 7fd778570640 1 -- 172.21.15.16:0/1356473091 wait complete. 2024-09-21T02:05:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:37 smithi016 ceph-mon[31245]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-09-21T02:05:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:37 smithi016 ceph-mon[31245]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:37.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-21T02:05:36.030809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655132,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":256,"num_read_kb":228,"num_write":369,"num_write_kb":2287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":393900,"kb_used_data":11372,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547924,"statfs":{"total":1151940427776,"available":1151537074176,"internally_reserved":0,"allocated":11644928,"data_stored":7707410,"data_compressed":393510,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47118,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":103,"num_write":31,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003892"},"pg_stats":[{"pgid":"2.1f","version":"84'13","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:30.758194+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:05:30.758194+0000","last_peered":"2024-09-21T02:05:30.758194+0000","last_clean":"2024-09-21T02:05:30.758194+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:05:30.758194+0000","last_undegraded":"2024-09-21T02:05:30.758194+0000","last_fullsized":"2024-09-21T02:05:30.758194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140454+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:04:54.140454+0000","last_peered":"2024-09-21T02:04:54.140454+0000","last_clean":"2024-09-21T02:04:54.140454+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:04:54.140454+0000","last_undegraded":"2024-09-21T02:04:54.140454+0000","last_fullsized":"2024-09-21T02:04:54.140454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737616+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:04:53.737616+0000","last_peered":"2024-09-21T02:04:53.737616+0000","last_clean":"2024-09-21T02:04:53.737616+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:04:53.737616+0000","last_undegraded":"2024-09-21T02:04:53.737616+0000","last_fullsized":"2024-09-21T02:04:53.737616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.142014+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:04:54.142014+0000","last_peered":"2024-09-21T02:04:54.142014+0000","last_clean":"2024-09-21T02:04:54.142014+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:04:54.142014+0000","last_undegraded":"2024-09-21T02:04:54.142014+0000","last_fullsized":"2024-09-21T02:04:54.142014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745962+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:04:53.745962+0000","last_peered":"2024-09-21T02:04:53.745962+0000","last_clean":"2024-09-21T02:04:53.745962+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:04:53.745962+0000","last_undegraded":"2024-09-21T02:04:53.745962+0000","last_fullsized":"2024-09-21T02:04:53.745962+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737915+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:04:53.737915+0000","last_peered":"2024-09-21T02:04:53.737915+0000","last_clean":"2024-09-21T02:04:53.737915+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:04:53.737915+0000","last_undegraded":"2024-09-21T02:04:53.737915+0000","last_fullsized":"2024-09-21T02:04:53.737915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.741060+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:20.741060+0000","last_peered":"2024-09-21T02:05:20.741060+0000","last_clean":"2024-09-21T02:05:20.741060+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:20.741060+0000","last_undegraded":"2024-09-21T02:05:20.741060+0000","last_fullsized":"2024-09-21T02:05:20.741060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141378+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:04:54.141378+0000","last_peered":"2024-09-21T02:04:54.141378+0000","last_clean":"2024-09-21T02:04:54.141378+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:04:54.141378+0000","last_undegraded":"2024-09-21T02:04:54.141378+0000","last_fullsized":"2024-09-21T02:04:54.141378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738771+0000","last_change":"2024-09-21T02:04:40.694714+0000","last_active":"2024-09-21T02:04:53.738771+0000","last_peered":"2024-09-21T02:04:53.738771+0000","last_clean":"2024-09-21T02:04:53.738771+0000","last_became_active":"2024-09-21T02:04:40.694146+0000","last_became_peered":"2024-09-21T02:04:40.694146+0000","last_unstale":"2024-09-21T02:04:53.738771+0000","last_undegraded":"2024-09-21T02:04:53.738771+0000","last_fullsized":"2024-09-21T02:04:53.738771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:58.001324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.846610+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:19.846610+0000","last_peered":"2024-09-21T02:05:19.846610+0000","last_clean":"2024-09-21T02:05:19.846610+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:19.846610+0000","last_undegraded":"2024-09-21T02:05:19.846610+0000","last_fullsized":"2024-09-21T02:05:19.846610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.143250+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:04:54.143250+0000","last_peered":"2024-09-21T02:04:54.143250+0000","last_clean":"2024-09-21T02:04:54.143250+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:04:54.143250+0000","last_undegraded":"2024-09-21T02:04:54.143250+0000","last_fullsized":"2024-09-21T02:04:54.143250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738122+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:04:53.738122+0000","last_peered":"2024-09-21T02:04:53.738122+0000","last_clean":"2024-09-21T02:04:53.738122+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:04:53.738122+0000","last_undegraded":"2024-09-21T02:04:53.738122+0000","last_fullsized":"2024-09-21T02:04:53.738122+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742775+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:04:53.742775+0000","last_peered":"2024-09-21T02:04:53.742775+0000","last_clean":"2024-09-21T02:04:53.742775+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:04:53.742775+0000","last_undegraded":"2024-09-21T02:04:53.742775+0000","last_fullsized":"2024-09-21T02:04:53.742775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":51,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:16.818463+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:16.818463+0000","last_peered":"2024-09-21T02:05:16.818463+0000","last_clean":"2024-09-21T02:05:16.818463+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:16.818463+0000","last_undegraded":"2024-09-21T02:05:16.818463+0000","last_fullsized":"2024-09-21T02:05:16.818463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738517+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:04:53.738517+0000","last_peered":"2024-09-21T02:04:53.738517+0000","last_clean":"2024-09-21T02:04:53.738517+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:04:53.738517+0000","last_undegraded":"2024-09-21T02:04:53.738517+0000","last_fullsized":"2024-09-21T02:04:53.738517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140418+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:04:54.140418+0000","last_peered":"2024-09-21T02:04:54.140418+0000","last_clean":"2024-09-21T02:04:54.140418+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:04:54.140418+0000","last_undegraded":"2024-09-21T02:04:54.140418+0000","last_fullsized":"2024-09-21T02:04:54.140418+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837247+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:05:18.837247+0000","last_peered":"2024-09-21T02:05:18.837247+0000","last_clean":"2024-09-21T02:05:18.837247+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:05:18.837247+0000","last_undegraded":"2024-09-21T02:05:18.837247+0000","last_fullsized":"2024-09-21T02:05:18.837247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738339+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:04:53.738339+0000","last_peered":"2024-09-21T02:04:53.738339+0000","last_clean":"2024-09-21T02:04:53.738339+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:04:53.738339+0000","last_undegraded":"2024-09-21T02:04:53.738339+0000","last_fullsized":"2024-09-21T02:04:53.738339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740710+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:04:53.740710+0000","last_peered":"2024-09-21T02:04:53.740710+0000","last_clean":"2024-09-21T02:04:53.740710+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:04:53.740710+0000","last_undegraded":"2024-09-21T02:04:53.740710+0000","last_fullsized":"2024-09-21T02:04:53.740710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843140+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:19.843140+0000","last_peered":"2024-09-21T02:05:19.843140+0000","last_clean":"2024-09-21T02:05:19.843140+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:19.843140+0000","last_undegraded":"2024-09-21T02:05:19.843140+0000","last_fullsized":"2024-09-21T02:05:19.843140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737973+0000","last_change":"2024-09-21T02:04:40.697250+0000","last_active":"2024-09-21T02:04:53.737973+0000","last_peered":"2024-09-21T02:04:53.737973+0000","last_clean":"2024-09-21T02:04:53.737973+0000","last_became_active":"2024-09-21T02:04:40.696827+0000","last_became_peered":"2024-09-21T02:04:40.696827+0000","last_unstale":"2024-09-21T02:04:53.737973+0000","last_undegraded":"2024-09-21T02:04:53.737973+0000","last_fullsized":"2024-09-21T02:04:53.737973+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:37.617204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739384+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:04:53.739384+0000","last_peered":"2024-09-21T02:04:53.739384+0000","last_clean":"2024-09-21T02:04:53.739384+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:04:53.739384+0000","last_undegraded":"2024-09-21T02:04:53.739384+0000","last_fullsized":"2024-09-21T02:04:53.739384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:16.818469+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:16.818469+0000","last_peered":"2024-09-21T02:05:16.818469+0000","last_clean":"2024-09-21T02:05:16.818469+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:16.818469+0000","last_undegraded":"2024-09-21T02:05:16.818469+0000","last_fullsized":"2024-09-21T02:05:16.818469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738446+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:04:53.738446+0000","last_peered":"2024-09-21T02:04:53.738446+0000","last_clean":"2024-09-21T02:04:53.738446+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:04:53.738446+0000","last_undegraded":"2024-09-21T02:04:53.738446+0000","last_fullsized":"2024-09-21T02:04:53.738446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745495+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:04:53.745495+0000","last_peered":"2024-09-21T02:04:53.745495+0000","last_clean":"2024-09-21T02:04:53.745495+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:04:53.745495+0000","last_undegraded":"2024-09-21T02:04:53.745495+0000","last_fullsized":"2024-09-21T02:04:53.745495+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844755+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:19.844755+0000","last_peered":"2024-09-21T02:05:19.844755+0000","last_clean":"2024-09-21T02:05:19.844755+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:19.844755+0000","last_undegraded":"2024-09-21T02:05:19.844755+0000","last_fullsized":"2024-09-21T02:05:19.844755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737791+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:04:53.737791+0000","last_peered":"2024-09-21T02:04:53.737791+0000","last_clean":"2024-09-21T02:04:53.737791+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:04:53.737791+0000","last_undegraded":"2024-09-21T02:04:53.737791+0000","last_fullsized":"2024-09-21T02:04:53.737791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140817+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:04:54.140817+0000","last_peered":"2024-09-21T02:04:54.140817+0000","last_clean":"2024-09-21T02:04:54.140817+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:04:54.140817+0000","last_undegraded":"2024-09-21T02:04:54.140817+0000","last_fullsized":"2024-09-21T02:04:54.140817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'14","reported_seq":49,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:30.662284+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:05:30.662284+0000","last_peered":"2024-09-21T02:05:30.662284+0000","last_clean":"2024-09-21T02:05:30.662284+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:05:30.662284+0000","last_undegraded":"2024-09-21T02:05:30.662284+0000","last_fullsized":"2024-09-21T02:05:30.662284+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140165+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:04:54.140165+0000","last_peered":"2024-09-21T02:04:54.140165+0000","last_clean":"2024-09-21T02:04:54.140165+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:04:54.140165+0000","last_undegraded":"2024-09-21T02:04:54.140165+0000","last_fullsized":"2024-09-21T02:04:54.140165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141772+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:04:54.141772+0000","last_peered":"2024-09-21T02:04:54.141772+0000","last_clean":"2024-09-21T02:04:54.141772+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:04:54.141772+0000","last_undegraded":"2024-09-21T02:04:54.141772+0000","last_fullsized":"2024-09-21T02:04:54.141772+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845615+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:19.845615+0000","last_peered":"2024-09-21T02:05:19.845615+0000","last_clean":"2024-09-21T02:05:19.845615+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:19.845615+0000","last_undegraded":"2024-09-21T02:05:19.845615+0000","last_fullsized":"2024-09-21T02:05:19.845615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741256+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:04:53.741256+0000","last_peered":"2024-09-21T02:04:53.741256+0000","last_clean":"2024-09-21T02:04:53.741256+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:04:53.741256+0000","last_undegraded":"2024-09-21T02:04:53.741256+0000","last_fullsized":"2024-09-21T02:04:53.741256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741019+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:04:53.741019+0000","last_peered":"2024-09-21T02:04:53.741019+0000","last_clean":"2024-09-21T02:04:53.741019+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:04:53.741019+0000","last_undegraded":"2024-09-21T02:04:53.741019+0000","last_fullsized":"2024-09-21T02:04:53.741019+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":36,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737883+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:04:53.737883+0000","last_peered":"2024-09-21T02:04:53.737883+0000","last_clean":"2024-09-21T02:04:53.737883+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:04:53.737883+0000","last_undegraded":"2024-09-21T02:04:53.737883+0000","last_fullsized":"2024-09-21T02:04:53.737883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739717+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:04:53.739717+0000","last_peered":"2024-09-21T02:04:53.739717+0000","last_clean":"2024-09-21T02:04:53.739717+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:04:53.739717+0000","last_undegraded":"2024-09-21T02:04:53.739717+0000","last_fullsized":"2024-09-21T02:04:53.739717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740866+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:04:53.740866+0000","last_peered":"2024-09-21T02:04:53.740866+0000","last_clean":"2024-09-21T02:04:53.740866+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:04:53.740866+0000","last_undegraded":"2024-09-21T02:04:53.740866+0000","last_fullsized":"2024-09-21T02:04:53.740866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":59,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:34.764884+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:05:34.764884+0000","last_peered":"2024-09-21T02:05:34.764884+0000","last_clean":"2024-09-21T02:05:34.764884+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:05:34.764884+0000","last_undegraded":"2024-09-21T02:05:34.764884+0000","last_fullsized":"2024-09-21T02:05:34.764884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141410+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:04:54.141410+0000","last_peered":"2024-09-21T02:04:54.141410+0000","last_clean":"2024-09-21T02:04:54.141410+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:04:54.141410+0000","last_undegraded":"2024-09-21T02:04:54.141410+0000","last_fullsized":"2024-09-21T02:04:54.141410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744493+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:04:53.744493+0000","last_peered":"2024-09-21T02:04:53.744493+0000","last_clean":"2024-09-21T02:04:53.744493+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:04:53.744493+0000","last_undegraded":"2024-09-21T02:04:53.744493+0000","last_fullsized":"2024-09-21T02:04:53.744493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:34.766656+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:05:34.766656+0000","last_peered":"2024-09-21T02:05:34.766656+0000","last_clean":"2024-09-21T02:05:34.766656+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:05:34.766656+0000","last_undegraded":"2024-09-21T02:05:34.766656+0000","last_fullsized":"2024-09-21T02:05:34.766656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141685+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:04:54.141685+0000","last_peered":"2024-09-21T02:04:54.141685+0000","last_clean":"2024-09-21T02:04:54.141685+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:04:54.141685+0000","last_undegraded":"2024-09-21T02:04:54.141685+0000","last_fullsized":"2024-09-21T02:04:54.141685+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141256+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:04:54.141256+0000","last_peered":"2024-09-21T02:04:54.141256+0000","last_clean":"2024-09-21T02:04:54.141256+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:04:54.141256+0000","last_undegraded":"2024-09-21T02:04:54.141256+0000","last_fullsized":"2024-09-21T02:04:54.141256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:17.826289+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:17.826289+0000","last_peered":"2024-09-21T02:05:17.826289+0000","last_clean":"2024-09-21T02:05:17.826289+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:17.826289+0000","last_undegraded":"2024-09-21T02:05:17.826289+0000","last_fullsized":"2024-09-21T02:05:17.826289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745568+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:04:53.745568+0000","last_peered":"2024-09-21T02:04:53.745568+0000","last_clean":"2024-09-21T02:04:53.745568+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:04:53.745568+0000","last_undegraded":"2024-09-21T02:04:53.745568+0000","last_fullsized":"2024-09-21T02:04:53.745568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140709+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:04:54.140709+0000","last_peered":"2024-09-21T02:04:54.140709+0000","last_clean":"2024-09-21T02:04:54.140709+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:04:54.140709+0000","last_undegraded":"2024-09-21T02:04:54.140709+0000","last_fullsized":"2024-09-21T02:04:54.140709+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.740461+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:20.740461+0000","last_peered":"2024-09-21T02:05:20.740461+0000","last_clean":"2024-09-21T02:05:20.740461+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:20.740461+0000","last_undegraded":"2024-09-21T02:05:20.740461+0000","last_fullsized":"2024-09-21T02:05:20.740461+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":106,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743276+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:04:53.743276+0000","last_peered":"2024-09-21T02:04:53.743276+0000","last_clean":"2024-09-21T02:04:53.743276+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:04:53.743276+0000","last_undegraded":"2024-09-21T02:04:53.743276+0000","last_fullsized":"2024-09-21T02:04:53.743276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743350+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:04:53.743350+0000","last_peered":"2024-09-21T02:04:53.743350+0000","last_clean":"2024-09-21T02:04:53.743350+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:04:53.743350+0000","last_undegraded":"2024-09-21T02:04:53.743350+0000","last_fullsized":"2024-09-21T02:04:53.743350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743486+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:04:53.743486+0000","last_peered":"2024-09-21T02:04:53.743486+0000","last_clean":"2024-09-21T02:04:53.743486+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:04:53.743486+0000","last_undegraded":"2024-09-21T02:04:53.743486+0000","last_fullsized":"2024-09-21T02:04:53.743486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"84'6","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:29.758348+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:05:29.758348+0000","last_peered":"2024-09-21T02:05:29.758348+0000","last_clean":"2024-09-21T02:05:29.758348+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:05:29.758348+0000","last_undegraded":"2024-09-21T02:05:29.758348+0000","last_fullsized":"2024-09-21T02:05:29.758348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140997+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:04:54.140997+0000","last_peered":"2024-09-21T02:04:54.140997+0000","last_clean":"2024-09-21T02:04:54.140997+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:04:54.140997+0000","last_undegraded":"2024-09-21T02:04:54.140997+0000","last_fullsized":"2024-09-21T02:04:54.140997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.749586+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:04:53.749586+0000","last_peered":"2024-09-21T02:04:53.749586+0000","last_clean":"2024-09-21T02:04:53.749586+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:04:53.749586+0000","last_undegraded":"2024-09-21T02:04:53.749586+0000","last_fullsized":"2024-09-21T02:04:53.749586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844266+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:05:19.844266+0000","last_peered":"2024-09-21T02:05:19.844266+0000","last_clean":"2024-09-21T02:05:19.844266+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:05:19.844266+0000","last_undegraded":"2024-09-21T02:05:19.844266+0000","last_fullsized":"2024-09-21T02:05:19.844266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745226+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:04:53.745226+0000","last_peered":"2024-09-21T02:04:53.745226+0000","last_clean":"2024-09-21T02:04:53.745226+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:04:53.745226+0000","last_undegraded":"2024-09-21T02:04:53.745226+0000","last_fullsized":"2024-09-21T02:04:53.745226+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743449+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:04:53.743449+0000","last_peered":"2024-09-21T02:04:53.743449+0000","last_clean":"2024-09-21T02:04:53.743449+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:04:53.743449+0000","last_undegraded":"2024-09-21T02:04:53.743449+0000","last_fullsized":"2024-09-21T02:04:53.743449+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843840+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:19.843840+0000","last_peered":"2024-09-21T02:05:19.843840+0000","last_clean":"2024-09-21T02:05:19.843840+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:19.843840+0000","last_undegraded":"2024-09-21T02:05:19.843840+0000","last_fullsized":"2024-09-21T02:05:19.843840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744290+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:04:53.744290+0000","last_peered":"2024-09-21T02:04:53.744290+0000","last_clean":"2024-09-21T02:04:53.744290+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:04:53.744290+0000","last_undegraded":"2024-09-21T02:04:53.744290+0000","last_fullsized":"2024-09-21T02:04:53.744290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741744+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:04:53.741744+0000","last_peered":"2024-09-21T02:04:53.741744+0000","last_clean":"2024-09-21T02:04:53.741744+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:04:53.741744+0000","last_undegraded":"2024-09-21T02:04:53.741744+0000","last_fullsized":"2024-09-21T02:04:53.741744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.820677+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:05:19.820677+0000","last_peered":"2024-09-21T02:05:19.820677+0000","last_clean":"2024-09-21T02:05:19.820677+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:05:19.820677+0000","last_undegraded":"2024-09-21T02:05:19.820677+0000","last_fullsized":"2024-09-21T02:05:19.820677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745149+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:04:53.745149+0000","last_peered":"2024-09-21T02:04:53.745149+0000","last_clean":"2024-09-21T02:04:53.745149+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:04:53.745149+0000","last_undegraded":"2024-09-21T02:04:53.745149+0000","last_fullsized":"2024-09-21T02:04:53.745149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.143763+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:04:54.143763+0000","last_peered":"2024-09-21T02:04:54.143763+0000","last_clean":"2024-09-21T02:04:54.143763+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:04:54.143763+0000","last_undegraded":"2024-09-21T02:04:54.143763+0000","last_fullsized":"2024-09-21T02:04:54.143763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845181+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:19.845181+0000","last_peered":"2024-09-21T02:05:19.845181+0000","last_clean":"2024-09-21T02:05:19.845181+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:19.845181+0000","last_undegraded":"2024-09-21T02:05:19.845181+0000","last_fullsized":"2024-09-21T02:05:19.845181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141822+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:04:54.141822+0000","last_peered":"2024-09-21T02:04:54.141822+0000","last_clean":"2024-09-21T02:04:54.141822+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:04:54.141822+0000","last_undegraded":"2024-09-21T02:04:54.141822+0000","last_fullsized":"2024-09-21T02:04:54.141822+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742465+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:04:53.742465+0000","last_peered":"2024-09-21T02:04:53.742465+0000","last_clean":"2024-09-21T02:04:53.742465+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:04:53.742465+0000","last_undegraded":"2024-09-21T02:04:53.742465+0000","last_fullsized":"2024-09-21T02:04:53.742465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":57,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:31.763600+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:05:31.763600+0000","last_peered":"2024-09-21T02:05:31.763600+0000","last_clean":"2024-09-21T02:05:31.763600+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:05:31.763600+0000","last_undegraded":"2024-09-21T02:05:31.763600+0000","last_fullsized":"2024-09-21T02:05:31.763600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738022+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:04:53.738022+0000","last_peered":"2024-09-21T02:04:53.738022+0000","last_clean":"2024-09-21T02:04:53.738022+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:04:53.738022+0000","last_undegraded":"2024-09-21T02:04:53.738022+0000","last_fullsized":"2024-09-21T02:04:53.738022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742906+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:04:53.742906+0000","last_peered":"2024-09-21T02:04:53.742906+0000","last_clean":"2024-09-21T02:04:53.742906+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:04:53.742906+0000","last_undegraded":"2024-09-21T02:04:53.742906+0000","last_fullsized":"2024-09-21T02:04:53.742906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.844769+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:19.844769+0000","last_peered":"2024-09-21T02:05:19.844769+0000","last_clean":"2024-09-21T02:05:19.844769+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:19.844769+0000","last_undegraded":"2024-09-21T02:05:19.844769+0000","last_fullsized":"2024-09-21T02:05:19.844769+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741306+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:04:53.741306+0000","last_peered":"2024-09-21T02:04:53.741306+0000","last_clean":"2024-09-21T02:04:53.741306+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:04:53.741306+0000","last_undegraded":"2024-09-21T02:04:53.741306+0000","last_fullsized":"2024-09-21T02:04:53.741306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141420+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:04:54.141420+0000","last_peered":"2024-09-21T02:04:54.141420+0000","last_clean":"2024-09-21T02:04:54.141420+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:04:54.141420+0000","last_undegraded":"2024-09-21T02:04:54.141420+0000","last_fullsized":"2024-09-21T02:04:54.141420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":46,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:20.741159+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:20.741159+0000","last_peered":"2024-09-21T02:05:20.741159+0000","last_clean":"2024-09-21T02:05:20.741159+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:20.741159+0000","last_undegraded":"2024-09-21T02:05:20.741159+0000","last_fullsized":"2024-09-21T02:05:20.741159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744934+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:04:53.744934+0000","last_peered":"2024-09-21T02:04:53.744934+0000","last_clean":"2024-09-21T02:04:53.744934+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:04:53.744934+0000","last_undegraded":"2024-09-21T02:04:53.744934+0000","last_fullsized":"2024-09-21T02:04:53.744934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739572+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:04:53.739572+0000","last_peered":"2024-09-21T02:04:53.739572+0000","last_clean":"2024-09-21T02:04:53.739572+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:04:53.739572+0000","last_undegraded":"2024-09-21T02:04:53.739572+0000","last_fullsized":"2024-09-21T02:04:53.739572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'31","reported_seq":99,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.769477+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:05:32.769477+0000","last_peered":"2024-09-21T02:05:32.769477+0000","last_clean":"2024-09-21T02:05:32.769477+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:05:32.769477+0000","last_undegraded":"2024-09-21T02:05:32.769477+0000","last_fullsized":"2024-09-21T02:05:32.769477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56544,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":71,"num_write":33,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737182+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:04:53.737182+0000","last_peered":"2024-09-21T02:04:53.737182+0000","last_clean":"2024-09-21T02:04:53.737182+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:04:53.737182+0000","last_undegraded":"2024-09-21T02:04:53.737182+0000","last_fullsized":"2024-09-21T02:04:53.737182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141299+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:04:54.141299+0000","last_peered":"2024-09-21T02:04:54.141299+0000","last_clean":"2024-09-21T02:04:54.141299+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:04:54.141299+0000","last_undegraded":"2024-09-21T02:04:54.141299+0000","last_fullsized":"2024-09-21T02:04:54.141299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":41,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837715+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:18.837715+0000","last_peered":"2024-09-21T02:05:18.837715+0000","last_clean":"2024-09-21T02:05:18.837715+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:18.837715+0000","last_undegraded":"2024-09-21T02:05:18.837715+0000","last_fullsized":"2024-09-21T02:05:18.837715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738451+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:04:53.738451+0000","last_peered":"2024-09-21T02:04:53.738451+0000","last_clean":"2024-09-21T02:04:53.738451+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:04:53.738451+0000","last_undegraded":"2024-09-21T02:04:53.738451+0000","last_fullsized":"2024-09-21T02:04:53.738451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.746035+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:04:53.746035+0000","last_peered":"2024-09-21T02:04:53.746035+0000","last_clean":"2024-09-21T02:04:53.746035+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:04:53.746035+0000","last_undegraded":"2024-09-21T02:04:53.746035+0000","last_fullsized":"2024-09-21T02:04:53.746035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.845342+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:19.845342+0000","last_peered":"2024-09-21T02:05:19.845342+0000","last_clean":"2024-09-21T02:05:19.845342+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:19.845342+0000","last_undegraded":"2024-09-21T02:05:19.845342+0000","last_fullsized":"2024-09-21T02:05:19.845342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.749629+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:04:53.749629+0000","last_peered":"2024-09-21T02:04:53.749629+0000","last_clean":"2024-09-21T02:04:53.749629+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:04:53.749629+0000","last_undegraded":"2024-09-21T02:04:53.749629+0000","last_fullsized":"2024-09-21T02:04:53.749629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141370+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:04:54.141370+0000","last_peered":"2024-09-21T02:04:54.141370+0000","last_clean":"2024-09-21T02:04:54.141370+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:04:54.141370+0000","last_undegraded":"2024-09-21T02:04:54.141370+0000","last_fullsized":"2024-09-21T02:04:54.141370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.761998+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:05:32.761998+0000","last_peered":"2024-09-21T02:05:32.761998+0000","last_clean":"2024-09-21T02:05:32.761998+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:05:32.761998+0000","last_undegraded":"2024-09-21T02:05:32.761998+0000","last_fullsized":"2024-09-21T02:05:32.761998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140370+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:04:54.140370+0000","last_peered":"2024-09-21T02:04:54.140370+0000","last_clean":"2024-09-21T02:04:54.140370+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:04:54.140370+0000","last_undegraded":"2024-09-21T02:04:54.140370+0000","last_fullsized":"2024-09-21T02:04:54.140370+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745297+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:04:53.745297+0000","last_peered":"2024-09-21T02:04:53.745297+0000","last_clean":"2024-09-21T02:04:53.745297+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:04:53.745297+0000","last_undegraded":"2024-09-21T02:04:53.745297+0000","last_fullsized":"2024-09-21T02:04:53.745297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737177+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:04:53.737177+0000","last_peered":"2024-09-21T02:04:53.737177+0000","last_clean":"2024-09-21T02:04:53.737177+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:04:53.737177+0000","last_undegraded":"2024-09-21T02:04:53.737177+0000","last_fullsized":"2024-09-21T02:04:53.737177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:19.843775+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:19.843775+0000","last_peered":"2024-09-21T02:05:19.843775+0000","last_clean":"2024-09-21T02:05:19.843775+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:19.843775+0000","last_undegraded":"2024-09-21T02:05:19.843775+0000","last_fullsized":"2024-09-21T02:05:19.843775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741573+0000","last_change":"2024-09-21T02:04:40.695415+0000","last_active":"2024-09-21T02:04:53.741573+0000","last_peered":"2024-09-21T02:04:53.741573+0000","last_clean":"2024-09-21T02:04:53.741573+0000","last_became_active":"2024-09-21T02:04:40.694905+0000","last_became_peered":"2024-09-21T02:04:40.694905+0000","last_unstale":"2024-09-21T02:04:53.741573+0000","last_undegraded":"2024-09-21T02:04:53.741573+0000","last_fullsized":"2024-09-21T02:04:53.741573+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:40.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140494+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:04:54.140494+0000","last_peered":"2024-09-21T02:04:54.140494+0000","last_clean":"2024-09-21T02:04:54.140494+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:04:54.140494+0000","last_undegraded":"2024-09-21T02:04:54.140494+0000","last_fullsized":"2024-09-21T02:04:54.140494+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:17.825059+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:17.825059+0000","last_peered":"2024-09-21T02:05:17.825059+0000","last_clean":"2024-09-21T02:05:17.825059+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:17.825059+0000","last_undegraded":"2024-09-21T02:05:17.825059+0000","last_fullsized":"2024-09-21T02:05:17.825059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:18.837433+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:18.837433+0000","last_peered":"2024-09-21T02:05:18.837433+0000","last_clean":"2024-09-21T02:05:18.837433+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:18.837433+0000","last_undegraded":"2024-09-21T02:05:18.837433+0000","last_fullsized":"2024-09-21T02:05:18.837433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.142556+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:04:54.142556+0000","last_peered":"2024-09-21T02:04:54.142556+0000","last_clean":"2024-09-21T02:04:54.142556+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:04:54.142556+0000","last_undegraded":"2024-09-21T02:04:54.142556+0000","last_fullsized":"2024-09-21T02:04:54.142556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738807+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:04:53.738807+0000","last_peered":"2024-09-21T02:04:53.738807+0000","last_clean":"2024-09-21T02:04:53.738807+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:04:53.738807+0000","last_undegraded":"2024-09-21T02:04:53.738807+0000","last_fullsized":"2024-09-21T02:04:53.738807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'6","reported_seq":58,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:05:32.762793+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:05:32.762793+0000","last_peered":"2024-09-21T02:05:32.762793+0000","last_clean":"2024-09-21T02:05:32.762793+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:05:32.762793+0000","last_undegraded":"2024-09-21T02:05:32.762793+0000","last_fullsized":"2024-09-21T02:05:32.762793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739925+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:04:53.739925+0000","last_peered":"2024-09-21T02:04:53.739925+0000","last_clean":"2024-09-21T02:04:53.739925+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:04:53.739925+0000","last_undegraded":"2024-09-21T02:04:53.739925+0000","last_fullsized":"2024-09-21T02:04:53.739925+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738584+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:04:53.738584+0000","last_peered":"2024-09-21T02:04:53.738584+0000","last_clean":"2024-09-21T02:04:53.738584+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:04:53.738584+0000","last_undegraded":"2024-09-21T02:04:53.738584+0000","last_fullsized":"2024-09-21T02:04:53.738584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740306+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:04:53.740306+0000","last_peered":"2024-09-21T02:04:53.740306+0000","last_clean":"2024-09-21T02:04:53.740306+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:04:53.740306+0000","last_undegraded":"2024-09-21T02:04:53.740306+0000","last_fullsized":"2024-09-21T02:04:53.740306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738227+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:04:53.738227+0000","last_peered":"2024-09-21T02:04:53.738227+0000","last_clean":"2024-09-21T02:04:53.738227+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:04:53.738227+0000","last_undegraded":"2024-09-21T02:04:53.738227+0000","last_fullsized":"2024-09-21T02:04:53.738227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738237+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:04:53.738237+0000","last_peered":"2024-09-21T02:04:53.738237+0000","last_clean":"2024-09-21T02:04:53.738237+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:04:53.738237+0000","last_undegraded":"2024-09-21T02:04:53.738237+0000","last_fullsized":"2024-09-21T02:04:53.738237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745630+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:04:53.745630+0000","last_peered":"2024-09-21T02:04:53.745630+0000","last_clean":"2024-09-21T02:04:53.745630+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:04:53.745630+0000","last_undegraded":"2024-09-21T02:04:53.745630+0000","last_fullsized":"2024-09-21T02:04:53.745630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141656+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:04:54.141656+0000","last_peered":"2024-09-21T02:04:54.141656+0000","last_clean":"2024-09-21T02:04:54.141656+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:04:54.141656+0000","last_undegraded":"2024-09-21T02:04:54.141656+0000","last_fullsized":"2024-09-21T02:04:54.141656+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742150+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:04:53.742150+0000","last_peered":"2024-09-21T02:04:53.742150+0000","last_clean":"2024-09-21T02:04:53.742150+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:04:53.742150+0000","last_undegraded":"2024-09-21T02:04:53.742150+0000","last_fullsized":"2024-09-21T02:04:53.742150+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738850+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:04:53.738850+0000","last_peered":"2024-09-21T02:04:53.738850+0000","last_clean":"2024-09-21T02:04:53.738850+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:04:53.738850+0000","last_undegraded":"2024-09-21T02:04:53.738850+0000","last_fullsized":"2024-09-21T02:04:53.738850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140692+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:04:54.140692+0000","last_peered":"2024-09-21T02:04:54.140692+0000","last_clean":"2024-09-21T02:04:54.140692+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:04:54.140692+0000","last_undegraded":"2024-09-21T02:04:54.140692+0000","last_fullsized":"2024-09-21T02:04:54.140692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.744457+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:04:53.744457+0000","last_peered":"2024-09-21T02:04:53.744457+0000","last_clean":"2024-09-21T02:04:53.744457+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:04:53.744457+0000","last_undegraded":"2024-09-21T02:04:53.744457+0000","last_fullsized":"2024-09-21T02:04:53.744457+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140564+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:04:54.140564+0000","last_peered":"2024-09-21T02:04:54.140564+0000","last_clean":"2024-09-21T02:04:54.140564+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:04:54.140564+0000","last_undegraded":"2024-09-21T02:04:54.140564+0000","last_fullsized":"2024-09-21T02:04:54.140564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141643+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:04:54.141643+0000","last_peered":"2024-09-21T02:04:54.141643+0000","last_clean":"2024-09-21T02:04:54.141643+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:04:54.141643+0000","last_undegraded":"2024-09-21T02:04:54.141643+0000","last_fullsized":"2024-09-21T02:04:54.141643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141681+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:04:54.141681+0000","last_peered":"2024-09-21T02:04:54.141681+0000","last_clean":"2024-09-21T02:04:54.141681+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:04:54.141681+0000","last_undegraded":"2024-09-21T02:04:54.141681+0000","last_fullsized":"2024-09-21T02:04:54.141681+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.741405+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:04:53.741405+0000","last_peered":"2024-09-21T02:04:53.741405+0000","last_clean":"2024-09-21T02:04:53.741405+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:04:53.741405+0000","last_undegraded":"2024-09-21T02:04:53.741405+0000","last_fullsized":"2024-09-21T02:04:53.741405+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.739214+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:04:53.739214+0000","last_peered":"2024-09-21T02:04:53.739214+0000","last_clean":"2024-09-21T02:04:53.739214+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:04:53.739214+0000","last_undegraded":"2024-09-21T02:04:53.739214+0000","last_fullsized":"2024-09-21T02:04:53.739214+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.737539+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:04:53.737539+0000","last_peered":"2024-09-21T02:04:53.737539+0000","last_clean":"2024-09-21T02:04:53.737539+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:04:53.737539+0000","last_undegraded":"2024-09-21T02:04:53.737539+0000","last_fullsized":"2024-09-21T02:04:53.737539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742143+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:04:53.742143+0000","last_peered":"2024-09-21T02:04:53.742143+0000","last_clean":"2024-09-21T02:04:53.742143+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:04:53.742143+0000","last_undegraded":"2024-09-21T02:04:53.742143+0000","last_fullsized":"2024-09-21T02:04:53.742143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141821+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:04:54.141821+0000","last_peered":"2024-09-21T02:04:54.141821+0000","last_clean":"2024-09-21T02:04:54.141821+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:04:54.141821+0000","last_undegraded":"2024-09-21T02:04:54.141821+0000","last_fullsized":"2024-09-21T02:04:54.141821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745028+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:04:53.745028+0000","last_peered":"2024-09-21T02:04:53.745028+0000","last_clean":"2024-09-21T02:04:53.745028+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:04:53.745028+0000","last_undegraded":"2024-09-21T02:04:53.745028+0000","last_fullsized":"2024-09-21T02:04:53.745028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745118+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:04:53.745118+0000","last_peered":"2024-09-21T02:04:53.745118+0000","last_clean":"2024-09-21T02:04:53.745118+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:04:53.745118+0000","last_undegraded":"2024-09-21T02:04:53.745118+0000","last_fullsized":"2024-09-21T02:04:53.745118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141534+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:04:54.141534+0000","last_peered":"2024-09-21T02:04:54.141534+0000","last_clean":"2024-09-21T02:04:54.141534+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:04:54.141534+0000","last_undegraded":"2024-09-21T02:04:54.141534+0000","last_fullsized":"2024-09-21T02:04:54.141534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738569+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:04:53.738569+0000","last_peered":"2024-09-21T02:04:53.738569+0000","last_clean":"2024-09-21T02:04:53.738569+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:04:53.738569+0000","last_undegraded":"2024-09-21T02:04:53.738569+0000","last_fullsized":"2024-09-21T02:04:53.738569+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743491+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:04:53.743491+0000","last_peered":"2024-09-21T02:04:53.743491+0000","last_clean":"2024-09-21T02:04:53.743491+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:04:53.743491+0000","last_undegraded":"2024-09-21T02:04:53.743491+0000","last_fullsized":"2024-09-21T02:04:53.743491+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.748540+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:04:53.748540+0000","last_peered":"2024-09-21T02:04:53.748540+0000","last_clean":"2024-09-21T02:04:53.748540+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:04:53.748540+0000","last_undegraded":"2024-09-21T02:04:53.748540+0000","last_fullsized":"2024-09-21T02:04:53.748540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.141785+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:04:54.141785+0000","last_peered":"2024-09-21T02:04:54.141785+0000","last_clean":"2024-09-21T02:04:54.141785+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:04:54.141785+0000","last_undegraded":"2024-09-21T02:04:54.141785+0000","last_fullsized":"2024-09-21T02:04:54.141785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:54.140368+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:04:54.140368+0000","last_peered":"2024-09-21T02:04:54.140368+0000","last_clean":"2024-09-21T02:04:54.140368+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:04:54.140368+0000","last_undegraded":"2024-09-21T02:04:54.140368+0000","last_fullsized":"2024-09-21T02:04:54.140368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742612+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:04:53.742612+0000","last_peered":"2024-09-21T02:04:53.742612+0000","last_clean":"2024-09-21T02:04:53.742612+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:04:53.742612+0000","last_undegraded":"2024-09-21T02:04:53.742612+0000","last_fullsized":"2024-09-21T02:04:53.742612+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.740817+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:04:53.740817+0000","last_peered":"2024-09-21T02:04:53.740817+0000","last_clean":"2024-09-21T02:04:53.740817+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:04:53.740817+0000","last_undegraded":"2024-09-21T02:04:53.740817+0000","last_fullsized":"2024-09-21T02:04:53.740817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.743244+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:04:53.743244+0000","last_peered":"2024-09-21T02:04:53.743244+0000","last_clean":"2024-09-21T02:04:53.743244+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:04:53.743244+0000","last_undegraded":"2024-09-21T02:04:53.743244+0000","last_fullsized":"2024-09-21T02:04:53.743244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.738442+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:04:53.738442+0000","last_peered":"2024-09-21T02:04:53.738442+0000","last_clean":"2024-09-21T02:04:53.738442+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:04:53.738442+0000","last_undegraded":"2024-09-21T02:04:53.738442+0000","last_fullsized":"2024-09-21T02:04:53.738442+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745107+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:04:53.745107+0000","last_peered":"2024-09-21T02:04:53.745107+0000","last_clean":"2024-09-21T02:04:53.745107+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:04:53.745107+0000","last_undegraded":"2024-09-21T02:04:53.745107+0000","last_fullsized":"2024-09-21T02:04:53.745107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.745391+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:04:53.745391+0000","last_peered":"2024-09-21T02:04:53.745391+0000","last_clean":"2024-09-21T02:04:53.745391+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:04:53.745391+0000","last_undegraded":"2024-09-21T02:04:53.745391+0000","last_fullsized":"2024-09-21T02:04:53.745391+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-21T02:04:53.742011+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:04:53.742011+0000","last_peered":"2024-09-21T02:04:53.742011+0000","last_clean":"2024-09-21T02:04:53.742011+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:04:53.742011+0000","last_undegraded":"2024-09-21T02:04:53.742011+0000","last_fullsized":"2024-09-21T02:04:53.742011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64530,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":175,"num_read_kb":159,"num_write":244,"num_write_kb":135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":193590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710745,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":933888,"data_stored":481620,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1124868,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037795,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":485143,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234024,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":481313,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397549,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":1078704,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561075,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":925696,"data_stored":476942,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":1073504,"data_compressed":60058,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":481016,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117058,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":476259,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378695,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":535015,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542220,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":534628,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":478398,"data_compressed":23704,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":59186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:05:37.462 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:05:37.462 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T02:05:37.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T02:05:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:37 smithi123 ceph-mon[33453]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-09-21T02:05:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:37 smithi123 ceph-mon[33453]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:37.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:37 smithi161 ceph-mon[34061]: pgmap v304: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-09-21T02:05:37.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:37 smithi161 ceph-mon[34061]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:05:37.724 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 -- 172.21.15.16:0/2547136202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad80102a60 msgr2=0x7fad80102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 --2- 172.21.15.16:0/2547136202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad80102a60 0x7fad80102e70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fad70009900 tx=0x7fad7002a410 comp rx=0 tx=0).stop 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 -- 172.21.15.16:0/2547136202 shutdown_connections 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 --2- 172.21.15.16:0/2547136202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80108cf0 0x7fad8010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 --2- 172.21.15.16:0/2547136202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad801083a0 0x7fad801087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 --2- 172.21.15.16:0/2547136202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad80102a60 0x7fad80102e70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 -- 172.21.15.16:0/2547136202 >> 172.21.15.16:0/2547136202 conn(0x7fad800797f0 msgr2=0x7fad80079c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 -- 172.21.15.16:0/2547136202 shutdown_connections 2024-09-21T02:05:38.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.223+0000 7fad85d63640 1 -- 172.21.15.16:0/2547136202 wait complete. 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.224+0000 7fad85d63640 1 Processor -- start 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.224+0000 7fad85d63640 1 -- start start 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80102a60 0x7fad8019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad80108cf0 0x7fad8019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad80112860 con 0x7fad801083a0 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad801126e0 con 0x7fad80108cf0 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad85d63640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad801129e0 con 0x7fad80102a60 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56376/0 (socket says 172.21.15.16:56376) 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 -- 172.21.15.16:0/2786587629 learned_addr learned my addr 172.21.15.16:0/2786587629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7f7fe640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80102a60 0x7fad8019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7ffff640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad80108cf0 0x7fad8019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:38.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 -- 172.21.15.16:0/2786587629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80102a60 msgr2=0x7fad8019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80102a60 0x7fad8019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 -- 172.21.15.16:0/2786587629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad80108cf0 msgr2=0x7fad8019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad80108cf0 0x7fad8019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 -- 172.21.15.16:0/2786587629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad801a03b0 con 0x7fad801083a0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.225+0000 7fad7effd640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c970 tx=0x7fad7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.226+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad74007bb0 con 0x7fad801083a0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.226+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad74007d50 con 0x7fad801083a0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.226+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad801a06a0 con 0x7fad801083a0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.226+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad74004da0 con 0x7fad801083a0 2024-09-21T02:05:38.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.226+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801a6e00 con 0x7fad801083a0 2024-09-21T02:05:38.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.227+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad80103ce0 con 0x7fad801083a0 2024-09-21T02:05:38.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.228+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fad74004f60 con 0x7fad801083a0 2024-09-21T02:05:38.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.229+0000 7fad7cff9640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 0x7fad680b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:38.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.229+0000 7fad7f7fe640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 0x7fad680b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:38.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.229+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fad740e7b10 con 0x7fad801083a0 2024-09-21T02:05:38.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.230+0000 7fad7f7fe640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 0x7fad680b8b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad70000c00 tx=0x7fad70009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:38.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.232+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fad74014030 con 0x7fad801083a0 2024-09-21T02:05:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.496+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fad8019b570 con 0x7fad801083a0 2024-09-21T02:05:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.499+0000 7fad7cff9640 1 -- 172.21.15.16:0/2786587629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fad8019b570 con 0x7fad801083a0 2024-09-21T02:05:38.500 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-21T02:05:38.500 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDyKe5mATK1HRAA/VjQ3d8AAenQipvy17yhYg== 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 msgr2=0x7fad680b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 0x7fad680b8b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad70000c00 tx=0x7fad70009680 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 msgr2=0x7fad8019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c970 tx=0x7fad7400ce30 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 shutdown_connections 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fad680b6670 0x7fad680b8b80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad80108cf0 0x7fad8019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad801083a0 0x7fad8019a9e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 --2- 172.21.15.16:0/2786587629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad80102a60 0x7fad8019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:38.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 >> 172.21.15.16:0/2786587629 conn(0x7fad800797f0 msgr2=0x7fad80101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 shutdown_connections 2024-09-21T02:05:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:38.502+0000 7fad85d63640 1 -- 172.21.15.16:0/2786587629 wait complete. 2024-09-21T02:05:39.219 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:05:39.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T02:05:39.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T02:05:39.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T02:05:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:39 smithi123 ceph-mon[33453]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-21T02:05:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:05:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:05:39.558 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:39 smithi016 ceph-mon[31245]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-21T02:05:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:05:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:05:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:39 smithi161 ceph-mon[34061]: pgmap v305: 129 pgs: 129 active+clean; 640 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-21T02:05:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T02:05:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2786587629' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 -- 172.21.15.16:0/2040590349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 msgr2=0x7fe774103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 --2- 172.21.15.16:0/2040590349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe774103f80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe768009900 tx=0x7fe76802a410 comp rx=0 tx=0).stop 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 -- 172.21.15.16:0/2040590349 shutdown_connections 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 --2- 172.21.15.16:0/2040590349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 0x7fe774110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 --2- 172.21.15.16:0/2040590349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe7741098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 --2- 172.21.15.16:0/2040590349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe774103f80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 -- 172.21.15.16:0/2040590349 >> 172.21.15.16:0/2040590349 conn(0x7fe7740ff5e0 msgr2=0x7fe774101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 -- 172.21.15.16:0/2040590349 shutdown_connections 2024-09-21T02:05:40.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.034+0000 7fe77b20a640 1 -- 172.21.15.16:0/2040590349 wait complete. 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 Processor -- start 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 -- start start 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe77419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 0x7fe77419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe774113970 con 0x7fe774103b70 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe7741137f0 con 0x7fe7741094b0 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.035+0000 7fe77b20a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe774113af0 con 0x7fe774109e00 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56404/0 (socket says 172.21.15.16:56404) 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 -- 172.21.15.16:0/3844297380 learned_addr learned my addr 172.21.15.16:0/3844297380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe7797a7640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 0x7fe77419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:40.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe773fff640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe77419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 -- 172.21.15.16:0/3844297380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 msgr2=0x7fe77419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 0x7fe77419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 -- 172.21.15.16:0/3844297380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 msgr2=0x7fe77419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe77419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 -- 172.21.15.16:0/3844297380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7741a01f0 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe773fff640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe77419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe778fa6640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe768009a20 tx=0x7fe768004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76803f030 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7741a0480 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe7680035f0 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.036+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe768035070 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.037+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7741a09b0 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.037+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe774103f80 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.039+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fe768003aa0 con 0x7fe774103b70 2024-09-21T02:05:40.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.040+0000 7fe771ffb640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 0x7fe7640b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.040+0000 7fe773fff640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 0x7fe7640b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.040+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fe7681088a0 con 0x7fe774103b70 2024-09-21T02:05:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.041+0000 7fe773fff640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 0x7fe7640b8b30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe774108e80 tx=0x7fe760005d90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:40.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.044+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe7741a09b0 con 0x7fe774103b70 2024-09-21T02:05:40.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.235+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fe77419b470 con 0x7fe774103b70 2024-09-21T02:05:40.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.236+0000 7fe771ffb640 1 -- 172.21.15.16:0/3844297380 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v58) ==== 86+0+133 (secure 0 0 0) 0x7fe7680ceba0 con 0x7fe774103b70 2024-09-21T02:05:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 msgr2=0x7fe7640b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 0x7fe7640b8b30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe774108e80 tx=0x7fe760005d90 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 msgr2=0x7fe77419a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe768009a20 tx=0x7fe768004270 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 shutdown_connections 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe7640b6620 0x7fe7640b8b30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe774109e00 0x7fe77419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7741094b0 0x7fe77419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 --2- 172.21.15.16:0/3844297380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe774103b70 0x7fe77419a340 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 >> 172.21.15.16:0/3844297380 conn(0x7fe7740ff5e0 msgr2=0x7fe774101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 shutdown_connections 2024-09-21T02:05:40.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:40.239+0000 7fe77b20a640 1 -- 172.21.15.16:0/3844297380 wait complete. 2024-09-21T02:05:40.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:40 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3844297380' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:05:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:40 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3844297380' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:05:40.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:40 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3844297380' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T02:05:40.870 INFO:teuthology.orchestra.run.smithi016.stdout:[{"version":58,"timestamp":"2024-09-21T02:04:25.582155+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-09-21T02:05:40.870 INFO:tasks.ceph_manager:config epoch is 58 2024-09-21T02:05:40.870 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T02:05:40.871 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T02:05:40.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T02:05:41.130 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:41 smithi123 ceph-mon[33453]: pgmap v306: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-21T02:05:41.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:41 smithi016 ceph-mon[31245]: pgmap v306: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 -- 172.21.15.16:0/188086645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0108530 msgr2=0x7fb9b010ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 --2- 172.21.15.16:0/188086645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0108530 0x7fb9b010ed60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009ca0 tx=0x7fb9a802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 -- 172.21.15.16:0/188086645 shutdown_connections 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 --2- 172.21.15.16:0/188086645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0108530 0x7fb9b010ed60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 --2- 172.21.15.16:0/188086645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b0107be0 0x7fb9b0107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 --2- 172.21.15.16:0/188086645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b01022a0 0x7fb9b01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 -- 172.21.15.16:0/188086645 >> 172.21.15.16:0/188086645 conn(0x7fb9b0079ec0 msgr2=0x7fb9b007a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 -- 172.21.15.16:0/188086645 shutdown_connections 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.625+0000 7fb9b5655640 1 -- 172.21.15.16:0/188086645 wait complete. 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.626+0000 7fb9b5655640 1 Processor -- start 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.626+0000 7fb9b5655640 1 -- start start 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b01022a0 0x7fb9b019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b0108530 0x7fb9b019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b0113a60 con 0x7fb9b0107be0 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b01138e0 con 0x7fb9b0108530 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9b5655640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b0113be0 con 0x7fb9b01022a0 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:41.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56426/0 (socket says 172.21.15.16:56426) 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/944365855 learned_addr learned my addr 172.21.15.16:0/944365855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9aeffd640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b01022a0 0x7fb9b019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9af7fe640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b0108530 0x7fb9b019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/944365855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b01022a0 msgr2=0x7fb9b019a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b01022a0 0x7fb9b019a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/944365855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b0108530 msgr2=0x7fb9b019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b0108530 0x7fb9b019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 -- 172.21.15.16:0/944365855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b01a0260 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb9ae7fc640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400c8c0 tx=0x7fb9a400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.627+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4007bb0 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.628+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb9a4007d50 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.628+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b01a0550 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.628+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40186a0 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.628+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b01a6d70 con 0x7fb9b0107be0 2024-09-21T02:05:41.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.629+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b01026b0 con 0x7fb9b0107be0 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.631+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fb9a400fa00 con 0x7fb9b0107be0 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.632+0000 7fb997fff640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 0x7fb9980b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.632+0000 7fb9aeffd640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 0x7fb9980b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.632+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fb9a40e7cf0 con 0x7fb9b0107be0 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.633+0000 7fb9aeffd640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 0x7fb9980b8b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009bd0 tx=0x7fb9a0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:41.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.636+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb9a4014030 con 0x7fb9b0107be0 2024-09-21T02:05:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:41 smithi161 ceph-mon[34061]: pgmap v306: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-21T02:05:41.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.875+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb9b0109d90 con 0x7fb9b0107be0 2024-09-21T02:05:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.881+0000 7fb997fff640 1 -- 172.21.15.16:0/944365855 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+296454 (secure 0 0 0) 0x7fb9a40ae020 con 0x7fb9b0107be0 2024-09-21T02:05:41.886 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:41.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 msgr2=0x7fb9980b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:41.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 0x7fb9980b8b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009bd0 tx=0x7fb9a0005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 msgr2=0x7fb9b019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400c8c0 tx=0x7fb9a400cd80 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 shutdown_connections 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb9980b6620 0x7fb9980b8b30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b0108530 0x7fb9b019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9b0107be0 0x7fb9b019a950 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 --2- 172.21.15.16:0/944365855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9b01022a0 0x7fb9b019a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 >> 172.21.15.16:0/944365855 conn(0x7fb9b0079ec0 msgr2=0x7fb9b0103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 shutdown_connections 2024-09-21T02:05:41.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:41.885+0000 7fb9b5655640 1 -- 172.21.15.16:0/944365855 wait complete. 2024-09-21T02:05:42.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:42 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/944365855' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:05:42.623 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:42 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/944365855' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:05:42.653 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"flags":0,"active_gid":14223,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":3647438626},{"type":"v1","addr":"172.21.15.16:6801","nonce":3647438626}]},"active_addr":"172.21.15.16:6801/3647438626","active_change":"2024-09-21T01:57:51.923134+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24187,"name":"z","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24200,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3621112339}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2305818243}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4078492053}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1532412515}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3635314658}]}]} 2024-09-21T02:05:42.660 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T02:05:42.660 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T02:05:42.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:05:42.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:42 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/944365855' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T02:05:42.925 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.420+0000 7f6f7e065640 1 -- 172.21.15.16:0/1518822131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f78104a60 msgr2=0x7f6f78104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.420+0000 7f6f7e065640 1 --2- 172.21.15.16:0/1518822131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f78104a60 0x7f6f78104e70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6f68009900 tx=0x7f6f6802a410 comp rx=0 tx=0).stop 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 -- 172.21.15.16:0/1518822131 shutdown_connections 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 --2- 172.21.15.16:0/1518822131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 0x7f6f781103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 --2- 172.21.15.16:0/1518822131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f781053b0 0x7f6f7810d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 --2- 172.21.15.16:0/1518822131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f78104a60 0x7f6f78104e70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 -- 172.21.15.16:0/1518822131 >> 172.21.15.16:0/1518822131 conn(0x7f6f780796d0 msgr2=0x7f6f78079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 -- 172.21.15.16:0/1518822131 shutdown_connections 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 -- 172.21.15.16:0/1518822131 wait complete. 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.421+0000 7f6f7e065640 1 Processor -- start 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 -- start start 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78104a60 0x7f6f7819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 0x7f6f7819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781139d0 con 0x7f6f781053b0 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78113850 con 0x7f6f78104a60 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f7e065640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78113b50 con 0x7f6f7810dec0 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56446/0 (socket says 172.21.15.16:56446) 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f76ffd640 1 -- 172.21.15.16:0/3998239228 learned_addr learned my addr 172.21.15.16:0/3998239228 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f777fe640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78104a60 0x7f6f7819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.422+0000 7f6f77fff640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 0x7f6f7819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 -- 172.21.15.16:0/3998239228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 msgr2=0x7f6f7819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 0x7f6f7819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 -- 172.21.15.16:0/3998239228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78104a60 msgr2=0x7f6f7819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78104a60 0x7f6f7819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 -- 172.21.15.16:0/3998239228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f781a03d0 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f76ffd640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e9f0 tx=0x7f6f6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c00ccc0 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f6c0108c0 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f781a06c0 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c018750 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.423+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f781a6e20 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.424+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f78104e70 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.426+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6f6c010430 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.427+0000 7f6f74ff9640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 0x7f6f600b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.427+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f6f6c0e8160 con 0x7f6f781053b0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.427+0000 7f6f777fe640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 0x7f6f600b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.427+0000 7f6f777fe640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 0x7f6f600b8b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6f680057c0 tx=0x7f6f680058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.430+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6f6c0b2840 con 0x7f6f781053b0 2024-09-21T02:05:43.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:43 smithi123 ceph-mon[33453]: pgmap v307: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-09-21T02:05:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.619+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f781a7080 con 0x7f6f781053b0 2024-09-21T02:05:43.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:43 smithi016 ceph-mon[31245]: pgmap v307: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-09-21T02:05:43.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.622+0000 7f6f74ff9640 1 -- 172.21.15.16:0/3998239228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25057 (secure 0 0 0) 0x7f6f6c0b7780 con 0x7f6f781053b0 2024-09-21T02:05:43.623 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:43.623 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:53.733185+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:05:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 msgr2=0x7f6f600b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 0x7f6f600b8b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6f680057c0 tx=0x7f6f680058d0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 msgr2=0x7f6f7819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e9f0 tx=0x7f6f6c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 shutdown_connections 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6f600b65f0 0x7f6f600b8b00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f7810dec0 0x7f6f7819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f781053b0 0x7f6f7819aa00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 --2- 172.21.15.16:0/3998239228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78104a60 0x7f6f7819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 >> 172.21.15.16:0/3998239228 conn(0x7f6f780796d0 msgr2=0x7f6f781068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 shutdown_connections 2024-09-21T02:05:43.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:43.625+0000 7f6f7e065640 1 -- 172.21.15.16:0/3998239228 wait complete. 2024-09-21T02:05:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:43 smithi161 ceph-mon[34061]: pgmap v307: 129 pgs: 129 active+clean; 648 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-09-21T02:05:44.313 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T02:05:44.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:05:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:44.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:44 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3998239228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:44.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:44 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3998239228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:44.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:44 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3998239228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- 172.21.15.16:0/2009471974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b501083b0 msgr2=0x7f9b501087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 --2- 172.21.15.16:0/2009471974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b501083b0 0x7f9b501087c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009900 tx=0x7f9b4002a410 comp rx=0 tx=0).stop 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- 172.21.15.16:0/2009471974 shutdown_connections 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 --2- 172.21.15.16:0/2009471974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50108d00 0x7f9b5010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 --2- 172.21.15.16:0/2009471974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b501083b0 0x7f9b501087c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 --2- 172.21.15.16:0/2009471974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b50102a70 0x7f9b50102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- 172.21.15.16:0/2009471974 >> 172.21.15.16:0/2009471974 conn(0x7f9b50079710 msgr2=0x7f9b50079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- 172.21.15.16:0/2009471974 shutdown_connections 2024-09-21T02:05:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- 172.21.15.16:0/2009471974 wait complete. 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 Processor -- start 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.071+0000 7f9b58093640 1 -- start start 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50102a70 0x7f9b5019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b501083b0 0x7f9b5019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50113990 con 0x7f9b50108d00 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50113810 con 0x7f9b501083b0 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b58093640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50113b10 con 0x7f9b50102a70 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56462/0 (socket says 172.21.15.16:56462) 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 -- 172.21.15.16:0/2194899231 learned_addr learned my addr 172.21.15.16:0/2194899231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b5562e640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b501083b0 0x7f9b5019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b55e2f640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50102a70 0x7f9b5019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 -- 172.21.15.16:0/2194899231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50102a70 msgr2=0x7f9b5019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50102a70 0x7f9b5019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 -- 172.21.15.16:0/2194899231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b501083b0 msgr2=0x7f9b5019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b501083b0 0x7f9b5019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 -- 172.21.15.16:0/2194899231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b501a0270 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.072+0000 7f9b56630640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800cce0 tx=0x7f9b4800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.073+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4800f450 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.073+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b4800fa80 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.073+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b501a0560 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.073+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48018ad0 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.073+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b501a6d80 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.074+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50103cf0 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.075+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f9b4800f5f0 con 0x7f9b50108d00 2024-09-21T02:05:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.076+0000 7f9b46ffd640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 0x7f9b3c0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.076+0000 7f9b55e2f640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 0x7f9b3c0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.076+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f9b480e7ff0 con 0x7f9b50108d00 2024-09-21T02:05:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.077+0000 7f9b55e2f640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 0x7f9b3c0b8bb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00bbe0 tx=0x7f9b4c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.081+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9b48014030 con 0x7f9b50108d00 2024-09-21T02:05:45.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.269+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b50102e80 con 0x7f9b50108d00 2024-09-21T02:05:45.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.271+0000 7f9b46ffd640 1 -- 172.21.15.16:0/2194899231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+25057 (secure 0 0 0) 0x7f9b50102e80 con 0x7f9b50108d00 2024-09-21T02:05:45.272 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:05:45.272 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:04:53.733185+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 msgr2=0x7f9b3c0b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 0x7f9b3c0b8bb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00bbe0 tx=0x7f9b4c00b390 comp rx=0 tx=0).stop 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 msgr2=0x7f9b5019fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800cce0 tx=0x7f9b4800a620 comp rx=0 tx=0).stop 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 shutdown_connections 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9b3c0b66a0 0x7f9b3c0b8bb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b50108d00 0x7f9b5019fc00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b501083b0 0x7f9b5019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 --2- 172.21.15.16:0/2194899231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b50102a70 0x7f9b5019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 >> 172.21.15.16:0/2194899231 conn(0x7f9b50079710 msgr2=0x7f9b50104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 shutdown_connections 2024-09-21T02:05:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:45.274+0000 7f9b58093640 1 -- 172.21.15.16:0/2194899231 wait complete. 2024-09-21T02:05:45.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:45 smithi016 ceph-mon[31245]: pgmap v308: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-21T02:05:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:45 smithi161 ceph-mon[34061]: pgmap v308: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-21T02:05:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:45 smithi123 ceph-mon[33453]: pgmap v308: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-09-21T02:05:46.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T02:05:46.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T02:05:46.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T02:05:46.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T02:05:46.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T02:05:46.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T02:05:46.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T02:05:46.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T02:05:46.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.8 flush_pg_stats 2024-09-21T02:05:46.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.9 flush_pg_stats 2024-09-21T02:05:46.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.10 flush_pg_stats 2024-09-21T02:05:46.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell osd.11 flush_pg_stats 2024-09-21T02:05:46.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:46 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2194899231' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:46 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2194899231' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:46.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:46.678 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:46.681 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:46 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2194899231' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:05:46.789 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:46.791 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:47.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:47 smithi016 ceph-mon[31245]: pgmap v309: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-21T02:05:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:47 smithi161 ceph-mon[34061]: pgmap v309: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-21T02:05:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:47 smithi123 ceph-mon[33453]: pgmap v309: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-21T02:05:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:47.864 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:47.865 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:47.865 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 -- 172.21.15.16:0/4202633193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 msgr2=0x7ff7c8110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 --2- 172.21.15.16:0/4202633193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c8110310 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009ca0 tx=0x7ff7c002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 -- 172.21.15.16:0/4202633193 shutdown_connections 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 --2- 172.21.15.16:0/4202633193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c8110310 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 --2- 172.21.15.16:0/4202633193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 0x7ff7c810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 --2- 172.21.15.16:0/4202633193 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7c8104900 0x7ff7c8104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 -- 172.21.15.16:0/4202633193 >> 172.21.15.16:0/4202633193 conn(0x7ff7c8079720 msgr2=0x7ff7c8079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 -- 172.21.15.16:0/4202633193 shutdown_connections 2024-09-21T02:05:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.266+0000 7ff7ce766640 1 -- 172.21.15.16:0/4202633193 wait complete. 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 Processor -- start 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 -- start start 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7c8104900 0x7ff7c819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 0x7ff7c819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81139a0 con 0x7ff7c810deb0 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8113820 con 0x7ff7c8104900 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ce766640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8113b20 con 0x7ff7c8105250 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ccd03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ccd03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c819fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35506/0 (socket says 172.21.15.16:35506) 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7ccd03640 1 -- 172.21.15.16:0/3286136240 learned_addr learned my addr 172.21.15.16:0/3286136240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7c77fe640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 0x7ff7c819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.267+0000 7ff7c7fff640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7c8104900 0x7ff7c819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 -- 172.21.15.16:0/3286136240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 msgr2=0x7ff7c819a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 0x7ff7c819a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 -- 172.21.15.16:0/3286136240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7c8104900 msgr2=0x7ff7c819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7c8104900 0x7ff7c819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c81a02c0 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7c77fe640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7c8105250 0x7ff7c819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ccd03640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c819fb90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005cc0 tx=0x7ff7c0032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0032e00 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7c00044d0 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c81a0550 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0030b70 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.268+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c81a6d10 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.269+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7c8063b50 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.271+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7ff7c0030d10 con 0x7ff7c810deb0 2024-09-21T02:05:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.271+0000 7ff7c57fa640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7b00b65f0 0x7ff7b00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c7fff640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7b00b65f0 0x7ff7b00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7ff7c0108e70 con 0x7ff7c810deb0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c57fa640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] conn(0x7ff7b00c1980 0x7ff7b00c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7c003ce50 con 0x7ff7b00c1980 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7ff7c01092c0 con 0x7ff7c810deb0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c77fe640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] conn(0x7ff7b00c1980 0x7ff7b00c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c7fff640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7b00b65f0 0x7ff7b00b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800bbe0 tx=0x7ff7b800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.272+0000 7ff7c77fe640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] conn(0x7ff7b00c1980 0x7ff7b00c3e70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.273+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== osd.3 v2:172.21.15.16:6810/2441025500 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7c003ce50 con 0x7ff7b00c1980 2024-09-21T02:05:48.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.287+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 --> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7c8063fd0 con 0x7ff7b00c1980 2024-09-21T02:05:48.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.287+0000 7ff7c57fa640 1 -- 172.21.15.16:0/3286136240 <== osd.3 v2:172.21.15.16:6810/2441025500 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff7c8063fd0 con 0x7ff7b00c1980 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] conn(0x7ff7b00c1980 msgr2=0x7ff7b00c3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] conn(0x7ff7b00c1980 0x7ff7b00c3e70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7b00b65f0 msgr2=0x7ff7b00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff7b00b65f0 0x7ff7b00b8b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800bbe0 tx=0x7ff7b800b390 comp rx=0 tx=0).stop 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 msgr2=0x7ff7c819fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 --2- 172.21.15.16:0/3286136240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c810deb0 0x7ff7c819fb90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005cc0 tx=0x7ff7c0032a10 comp rx=0 tx=0).stop 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ccd03640 1 -- 172.21.15.16:0/3286136240 reap_dead start 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 shutdown_connections 2024-09-21T02:05:48.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 >> 172.21.15.16:0/3286136240 conn(0x7ff7c8079720 msgr2=0x7ff7c8100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:48.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 shutdown_connections 2024-09-21T02:05:48.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.288+0000 7ff7ce766640 1 -- 172.21.15.16:0/3286136240 wait complete. 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- 172.21.15.16:0/236207351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c108d10 msgr2=0x7fdb2c10f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 --2- 172.21.15.16:0/236207351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c108d10 0x7fdb2c10f540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009ca0 tx=0x7fdb2402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- 172.21.15.16:0/236207351 shutdown_connections 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 --2- 172.21.15.16:0/236207351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c108d10 0x7fdb2c10f540 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 --2- 172.21.15.16:0/236207351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 --2- 172.21.15.16:0/236207351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2c102a80 0x7fdb2c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- 172.21.15.16:0/236207351 >> 172.21.15.16:0/236207351 conn(0x7fdb2c079740 msgr2=0x7fdb2c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- 172.21.15.16:0/236207351 shutdown_connections 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- 172.21.15.16:0/236207351 wait complete. 2024-09-21T02:05:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 Processor -- start 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.484+0000 7fdb3211e640 1 -- start start 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2c102a80 0x7fdb2c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2c108d10 0x7fdb2c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c112880 con 0x7fdb2c1083c0 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c112700 con 0x7fdb2c102a80 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb3211e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c112a00 con 0x7fdb2c108d10 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35532/0 (socket says 172.21.15.16:35532) 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 -- 172.21.15.16:0/4269792305 learned_addr learned my addr 172.21.15.16:0/4269792305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb2b7fe640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2c102a80 0x7fdb2c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb2bfff640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2c108d10 0x7fdb2c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 -- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2c108d10 msgr2=0x7fdb2c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2c108d10 0x7fdb2c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 -- 172.21.15.16:0/4269792305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2c102a80 msgr2=0x7fdb2c19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2c102a80 0x7fdb2c19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.485+0000 7fdb22ffd640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c1a0390 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb22ffd640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c19a9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c970 tx=0x7fdb1400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14007bb0 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb14007d50 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c1a0680 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14004e10 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.486+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c1a6de0 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.487+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb2c063ad0 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.489+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fdb14005000 con 0x7fdb2c1083c0 2024-09-21T02:05:48.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.489+0000 7fdb297fa640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb180b65f0 0x7fdb180b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.489+0000 7fdb2b7fe640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb180b65f0 0x7fdb180b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.489+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fdb140e7ae0 con 0x7fdb2c1083c0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.490+0000 7fdb297fa640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] conn(0x7fdb180c1980 0x7fdb180c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.490+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb14012ec0 con 0x7fdb180c1980 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.490+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fdb140e7f30 con 0x7fdb2c1083c0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.490+0000 7fdb2b7fe640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb180b65f0 0x7fdb180b8b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00bbe0 tx=0x7fdb1c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.490+0000 7fdb2bfff640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] conn(0x7fdb180c1980 0x7fdb180c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.491+0000 7fdb2bfff640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] conn(0x7fdb180c1980 0x7fdb180c3e70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.492+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== osd.2 v2:172.21.15.161:6800/3882488819 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb14012ec0 con 0x7fdb180c1980 2024-09-21T02:05:48.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.505+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 --> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb2c063f50 con 0x7fdb180c1980 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb297fa640 1 -- 172.21.15.16:0/4269792305 <== osd.2 v2:172.21.15.161:6800/3882488819 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb2c063f50 con 0x7fdb180c1980 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] conn(0x7fdb180c1980 msgr2=0x7fdb180c3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] conn(0x7fdb180c1980 0x7fdb180c3e70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb180b65f0 msgr2=0x7fdb180b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb180b65f0 0x7fdb180b8b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00bbe0 tx=0x7fdb1c00b390 comp rx=0 tx=0).stop 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 msgr2=0x7fdb2c19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 --2- 172.21.15.16:0/4269792305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2c1083c0 0x7fdb2c19a9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c970 tx=0x7fdb1400ce30 comp rx=0 tx=0).stop 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb2bfff640 1 -- 172.21.15.16:0/4269792305 reap_dead start 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 shutdown_connections 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 >> 172.21.15.16:0/4269792305 conn(0x7fdb2c079740 msgr2=0x7fdb2c06bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 shutdown_connections 2024-09-21T02:05:48.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:48.506+0000 7fdb3211e640 1 -- 172.21.15.16:0/4269792305 wait complete. 2024-09-21T02:05:49.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:49 smithi016 ceph-mon[31245]: pgmap v310: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-09-21T02:05:49.465 INFO:teuthology.orchestra.run.smithi016.stdout:81604378699 2024-09-21T02:05:49.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T02:05:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1764192801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b81094b0 msgr2=0x7f01b81098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1764192801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b81094b0 0x7f01b81098c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009900 tx=0x7f01ac02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1764192801 shutdown_connections 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1764192801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 0x7f01b8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1764192801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b81094b0 0x7f01b81098c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1764192801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01b8103bc0 0x7f01b8103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1764192801 >> 172.21.15.16:0/1764192801 conn(0x7f01b807b910 msgr2=0x7f01b807bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1764192801 shutdown_connections 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.631+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1764192801 wait complete. 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 Processor -- start 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 -- start start 2024-09-21T02:05:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 0x7f01b819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01b81094b0 0x7f01b819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 0x7f01b819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01b8113970 con 0x7f01b8103bc0 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01b81137f0 con 0x7f01b8109e00 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01bc8b3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01b8113af0 con 0x7f01b81094b0 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01b659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 0x7f01b819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01b659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 0x7f01b819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35558/0 (socket says 172.21.15.16:35558) 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01b659c640 1 -- 172.21.15.16:0/1030112063 learned_addr learned my addr 172.21.15.16:0/1030112063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01b5d9b640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01b81094b0 0x7f01b819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.632+0000 7f01b6d9d640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 0x7f01b819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 -- 172.21.15.16:0/1030112063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01b81094b0 msgr2=0x7f01b819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01b81094b0 0x7f01b819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 -- 172.21.15.16:0/1030112063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 msgr2=0x7f01b819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 0x7f01b819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b81a03b0 con 0x7f01b8103bc0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b6d9d640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b8109e00 0x7f01b819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01b659c640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 0x7f01b819a440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c890 tx=0x7f01a800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8004220 con 0x7f01b8103bc0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b81a06a0 con 0x7f01b8103bc0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01a8007e40 con 0x7f01b8103bc0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8004a00 con 0x7f01b8103bc0 2024-09-21T02:05:49.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.633+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b81a6e00 con 0x7f01b8103bc0 2024-09-21T02:05:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.634+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f01b8063b50 con 0x7f01b8103bc0 2024-09-21T02:05:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.636+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f01a80079b0 con 0x7f01b8103bc0 2024-09-21T02:05:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01a77fe640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01a00b65f0 0x7f01a00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f01a80e7c10 con 0x7f01b8103bc0 2024-09-21T02:05:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01b5d9b640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01a00b65f0 0x7f01a00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01a77fe640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] conn(0x7f01a00c1980 0x7f01a00c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01a8011ec0 con 0x7f01a00c1980 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f01a80e8060 con 0x7f01b8103bc0 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.637+0000 7f01b6d9d640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] conn(0x7f01a00c1980 0x7f01a00c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.638+0000 7f01b5d9b640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01a00b65f0 0x7f01a00b8b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f01ac005940 tx=0x7f01ac0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.638+0000 7f01b6d9d640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] conn(0x7f01a00c1980 0x7f01a00c3e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.639+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== osd.0 v2:172.21.15.16:6802/1246966060 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01a8011ec0 con 0x7f01a00c1980 2024-09-21T02:05:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.653+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 --> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f01b8063f50 con 0x7f01a00c1980 2024-09-21T02:05:49.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.653+0000 7f01a77fe640 1 -- 172.21.15.16:0/1030112063 <== osd.0 v2:172.21.15.16:6802/1246966060 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f01b8063f50 con 0x7f01a00c1980 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.653+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] conn(0x7f01a00c1980 msgr2=0x7f01a00c3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] conn(0x7f01a00c1980 0x7f01a00c3e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01a00b65f0 msgr2=0x7f01a00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01a00b65f0 0x7f01a00b8b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f01ac005940 tx=0x7f01ac0058d0 comp rx=0 tx=0).stop 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 msgr2=0x7f01b819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 --2- 172.21.15.16:0/1030112063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b8103bc0 0x7f01b819a440 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c890 tx=0x7f01a800cd50 comp rx=0 tx=0).stop 2024-09-21T02:05:49.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01b6d9d640 1 -- 172.21.15.16:0/1030112063 reap_dead start 2024-09-21T02:05:49.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 shutdown_connections 2024-09-21T02:05:49.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 >> 172.21.15.16:0/1030112063 conn(0x7f01b807b910 msgr2=0x7f01b8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:49.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 shutdown_connections 2024-09-21T02:05:49.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:49.654+0000 7f01bc8b3640 1 -- 172.21.15.16:0/1030112063 wait complete. 2024-09-21T02:05:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:49 smithi161 ceph-mon[34061]: pgmap v310: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-09-21T02:05:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:49 smithi123 ceph-mon[33453]: pgmap v310: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 10 op/s 2024-09-21T02:05:50.333 INFO:teuthology.orchestra.run.smithi016.stdout:38654705749 2024-09-21T02:05:50.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2035256569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd741083d0 msgr2=0x7fcd741087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2035256569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd741083d0 0x7fcd741087e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcd64009900 tx=0x7fcd6402a410 comp rx=0 tx=0).stop 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2035256569 shutdown_connections 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2035256569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd74108d20 0x7fcd7410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2035256569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd741083d0 0x7fcd741087e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2035256569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd74102ae0 0x7fcd74102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2035256569 >> 172.21.15.16:0/2035256569 conn(0x7fcd740797b0 msgr2=0x7fcd74079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2035256569 shutdown_connections 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.503+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2035256569 wait complete. 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 Processor -- start 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 -- start start 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 0x7fcd74199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd741083d0 0x7fcd741998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd74108d20 0x7fcd7419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741128d0 con 0x7fcd74102ae0 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd74112750 con 0x7fcd741083d0 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.504+0000 7fcd7c6f8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcd74112a50 con 0x7fcd74108d20 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 0x7fcd74199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:50.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 0x7fcd74199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35586/0 (socket says 172.21.15.16:35586) 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 -- 172.21.15.16:0/2223012675 learned_addr learned my addr 172.21.15.16:0/2223012675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd79c93640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd741083d0 0x7fcd741998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7ac95640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd74108d20 0x7fcd7419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 -- 172.21.15.16:0/2223012675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd74108d20 msgr2=0x7fcd7419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd74108d20 0x7fcd7419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 -- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd741083d0 msgr2=0x7fcd741998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd741083d0 0x7fcd741998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7419f2b0 con 0x7fcd74102ae0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd79c93640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd741083d0 0x7fcd741998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7a494640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 0x7fcd74199340 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcd700078f0 tx=0x7fcd70007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70019020 con 0x7fcd74102ae0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7419f5a0 con 0x7fcd74102ae0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcd700054b0 con 0x7fcd74102ae0 2024-09-21T02:05:50.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.505+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000e630 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.506+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7406b820 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.506+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcd7406bd30 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.508+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fcd700046c0 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd6b7fe640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcd600b65f0 0x7fcd600b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fcd700e9d90 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd79c93640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcd600b65f0 0x7fcd600b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd6b7fe640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] conn(0x7fcd600c1980 0x7fcd600c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd7000ae50 con 0x7fcd600c1980 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.509+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fcd700ea1e0 con 0x7fcd74102ae0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.510+0000 7fcd7ac95640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] conn(0x7fcd600c1980 0x7fcd600c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.510+0000 7fcd79c93640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcd600b65f0 0x7fcd600b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd640057c0 tx=0x7fcd64009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:50.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.510+0000 7fcd7ac95640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] conn(0x7fcd600c1980 0x7fcd600c3e70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:50.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.511+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== osd.1 v2:172.21.15.123:6800/4233413393 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcd7000ae50 con 0x7fcd600c1980 2024-09-21T02:05:50.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.525+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 --> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcd7406c130 con 0x7fcd600c1980 2024-09-21T02:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.526+0000 7fcd6b7fe640 1 -- 172.21.15.16:0/2223012675 <== osd.1 v2:172.21.15.123:6800/4233413393 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcd7406c130 con 0x7fcd600c1980 2024-09-21T02:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] conn(0x7fcd600c1980 msgr2=0x7fcd600c3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] conn(0x7fcd600c1980 0x7fcd600c3e70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcd600b65f0 msgr2=0x7fcd600b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcd600b65f0 0x7fcd600b8b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd640057c0 tx=0x7fcd64009680 comp rx=0 tx=0).stop 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 msgr2=0x7fcd74199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 --2- 172.21.15.16:0/2223012675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd74102ae0 0x7fcd74199340 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcd700078f0 tx=0x7fcd70007db0 comp rx=0 tx=0).stop 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7ac95640 1 -- 172.21.15.16:0/2223012675 reap_dead start 2024-09-21T02:05:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 shutdown_connections 2024-09-21T02:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 >> 172.21.15.16:0/2223012675 conn(0x7fcd740797b0 msgr2=0x7fcd741044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 shutdown_connections 2024-09-21T02:05:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:50.527+0000 7fcd7c6f8640 1 -- 172.21.15.16:0/2223012675 wait complete. 2024-09-21T02:05:51.224 INFO:teuthology.orchestra.run.smithi016.stdout:115964117062 2024-09-21T02:05:51.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T02:05:51.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:51 smithi016 ceph-mon[31245]: pgmap v311: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-21T02:05:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/2153940885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc1062e0 msgr2=0x7f9fdc1066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/2153940885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc1062e0 0x7f9fdc1066f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009900 tx=0x7f9fd002a410 comp rx=0 tx=0).stop 2024-09-21T02:05:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/2153940885 shutdown_connections 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/2153940885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fdc106c30 0x7f9fdc110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/2153940885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc1062e0 0x7f9fdc1066f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/2153940885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fdc102000 0x7f9fdc102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/2153940885 >> 172.21.15.16:0/2153940885 conn(0x7f9fdc079e40 msgr2=0x7f9fdc07a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/2153940885 shutdown_connections 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.392+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/2153940885 wait complete. 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.393+0000 7f9fe0ed9640 1 Processor -- start 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.393+0000 7f9fe0ed9640 1 -- start start 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fdc102000 0x7f9fdc102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fdc1062e0 0x7f9fdc1034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 0x7f9fdc103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc113980 con 0x7f9fdc106c30 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc113800 con 0x7f9fdc102000 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fe0ed9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc113b00 con 0x7f9fdc1062e0 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 0x7f9fdc103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 0x7f9fdc103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35490/0 (socket says 172.21.15.16:35490) 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 -- 172.21.15.16:0/140445818 learned_addr learned my addr 172.21.15.16:0/140445818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fd9d9b640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fdc1062e0 0x7f9fdc1034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fda59c640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fdc102000 0x7f9fdc102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 -- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fdc1062e0 msgr2=0x7f9fdc1034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9fdc1062e0 0x7f9fdc1034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 -- 172.21.15.16:0/140445818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fdc102000 msgr2=0x7f9fdc102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fdc102000 0x7f9fdc102fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.394+0000 7f9fdad9d640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc1a8640 con 0x7f9fdc106c30 2024-09-21T02:05:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fdad9d640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 0x7f9fdc103a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400eaa0 tx=0x7f9fd400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4004d20 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9fd4005020 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc1a8930 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40116d0 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.395+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc1a8e40 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.396+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9fdc063b50 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.397+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f9fd40051d0 con 0x7f9fdc106c30 2024-09-21T02:05:51.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fcb7fe640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9fc40b6670 0x7f9fc40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fda59c640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9fc40b6670 0x7f9fc40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7f9fd40e7eb0 con 0x7f9fdc106c30 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fcb7fe640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] conn(0x7f9fc40c1a00 0x7f9fc40c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fd4014ec0 con 0x7f9fc40c1a00 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.398+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f9fd40e8300 con 0x7f9fdc106c30 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.399+0000 7f9fda59c640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9fc40b6670 0x7f9fc40b8b80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc1049f0 tx=0x7f9fcc00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.399+0000 7f9fd9d9b640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] conn(0x7f9fc40c1a00 0x7f9fc40c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.399+0000 7f9fd9d9b640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] conn(0x7f9fc40c1a00 0x7f9fc40c3ef0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.400+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== osd.5 v2:172.21.15.161:6808/1422110321 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9fd4014ec0 con 0x7f9fc40c1a00 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.414+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 --> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9fdc063f80 con 0x7f9fc40c1a00 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fcb7fe640 1 -- 172.21.15.16:0/140445818 <== osd.5 v2:172.21.15.161:6808/1422110321 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9fdc063f80 con 0x7f9fc40c1a00 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] conn(0x7f9fc40c1a00 msgr2=0x7f9fc40c3ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] conn(0x7f9fc40c1a00 0x7f9fc40c3ef0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9fc40b6670 msgr2=0x7f9fc40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9fc40b6670 0x7f9fc40b8b80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc1049f0 tx=0x7f9fcc00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 msgr2=0x7f9fdc103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 --2- 172.21.15.16:0/140445818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9fdc106c30 0x7f9fdc103a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400eaa0 tx=0x7f9fd400eea0 comp rx=0 tx=0).stop 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fdad9d640 1 -- 172.21.15.16:0/140445818 reap_dead start 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 shutdown_connections 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 >> 172.21.15.16:0/140445818 conn(0x7f9fdc079e40 msgr2=0x7f9fdc107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 shutdown_connections 2024-09-21T02:05:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:47.415+0000 7f9fe0ed9640 1 -- 172.21.15.16:0/140445818 wait complete. 2024-09-21T02:05:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:51 smithi161 ceph-mon[34061]: pgmap v311: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-21T02:05:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:51 smithi123 ceph-mon[33453]: pgmap v311: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-21T02:05:52.216 INFO:teuthology.orchestra.run.smithi016.stdout:163208757307 2024-09-21T02:05:52.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 -- 172.21.15.16:0/913195819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c10de10 msgr2=0x7fbb6c110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 --2- 172.21.15.16:0/913195819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c10de10 0x7fbb6c110320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009ca0 tx=0x7fbb6402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 -- 172.21.15.16:0/913195819 shutdown_connections 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 --2- 172.21.15.16:0/913195819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c10de10 0x7fbb6c110320 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 --2- 172.21.15.16:0/913195819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb6c105290 0x7fbb6c10d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 --2- 172.21.15.16:0/913195819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c104860 0x7fbb6c104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 -- 172.21.15.16:0/913195819 >> 172.21.15.16:0/913195819 conn(0x7fbb6c079740 msgr2=0x7fbb6c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:52.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 -- 172.21.15.16:0/913195819 shutdown_connections 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 -- 172.21.15.16:0/913195819 wait complete. 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.759+0000 7fbb747d9640 1 Processor -- start 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 -- start start 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 0x7fbb6c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c105290 0x7fbb6c19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb6c10de10 0x7fbb6c19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c113a30 con 0x7fbb6c104860 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c1138b0 con 0x7fbb6c10de10 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb747d9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c113bb0 con 0x7fbb6c105290 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 0x7fbb6c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 0x7fbb6c19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35608/0 (socket says 172.21.15.16:35608) 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 -- 172.21.15.16:0/1672888099 learned_addr learned my addr 172.21.15.16:0/1672888099 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:52.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72d76640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb6c10de10 0x7fbb6c19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb71d74640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c105290 0x7fbb6c19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 -- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c105290 msgr2=0x7fbb6c19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c105290 0x7fbb6c19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 -- 172.21.15.16:0/1672888099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb6c10de10 msgr2=0x7fbb6c19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb6c10de10 0x7fbb6c19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb72575640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c1a03f0 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.760+0000 7fbb71d74640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb6c105290 0x7fbb6c19aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb72575640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 0x7fbb6c19a480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800b450 tx=0x7fbb6800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68015020 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb68011d20 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6c1a06e0 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68010e10 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.761+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6c102650 con 0x7fbb6c104860 2024-09-21T02:05:52.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.762+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb6c1083f0 con 0x7fbb6c104860 2024-09-21T02:05:52.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.763+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fbb68018020 con 0x7fbb6c104860 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb637fe640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb580b65f0 0x7fbb580b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9307+0+0 (secure 0 0 0) 0x7fbb680e7370 con 0x7fbb6c104860 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb71d74640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb580b65f0 0x7fbb580b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb637fe640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] conn(0x7fbb580c1980 0x7fbb580c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb68018ec0 con 0x7fbb580c1980 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.764+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fbb680ee050 con 0x7fbb6c104860 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.765+0000 7fbb71d74640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb580b65f0 0x7fbb580b8b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c10fa90 tx=0x7fbb5c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.765+0000 7fbb72d76640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] conn(0x7fbb580c1980 0x7fbb580c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.765+0000 7fbb72d76640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] conn(0x7fbb580c1980 0x7fbb580c3e70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:52.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.766+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== osd.8 v2:172.21.15.161:6816/3772925422 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb68018ec0 con 0x7fbb580c1980 2024-09-21T02:05:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.780+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 --> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb6c06b390 con 0x7fbb580c1980 2024-09-21T02:05:52.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb637fe640 1 -- 172.21.15.16:0/1672888099 <== osd.8 v2:172.21.15.161:6816/3772925422 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbb6c06b390 con 0x7fbb580c1980 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] conn(0x7fbb580c1980 msgr2=0x7fbb580c3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] conn(0x7fbb580c1980 0x7fbb580c3e70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb580b65f0 msgr2=0x7fbb580b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb580b65f0 0x7fbb580b8b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c10fa90 tx=0x7fbb5c005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 msgr2=0x7fbb6c19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:52.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 --2- 172.21.15.16:0/1672888099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c104860 0x7fbb6c19a480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800b450 tx=0x7fbb6800b910 comp rx=0 tx=0).stop 2024-09-21T02:05:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb72d76640 1 -- 172.21.15.16:0/1672888099 reap_dead start 2024-09-21T02:05:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 shutdown_connections 2024-09-21T02:05:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 >> 172.21.15.16:0/1672888099 conn(0x7fbb6c079740 msgr2=0x7fbb6c1079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 shutdown_connections 2024-09-21T02:05:52.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:52.781+0000 7fbb747d9640 1 -- 172.21.15.16:0/1672888099 wait complete. 2024-09-21T02:05:53.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:53 smithi016 ceph-mon[31245]: pgmap v312: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T02:05:53.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]} : dispatch 2024-09-21T02:05:53.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]} : dispatch 2024-09-21T02:05:53.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T02:05:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:53 smithi161 ceph-mon[34061]: pgmap v312: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T02:05:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]} : dispatch 2024-09-21T02:05:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]} : dispatch 2024-09-21T02:05:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T02:05:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:53 smithi123 ceph-mon[33453]: pgmap v312: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-09-21T02:05:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]} : dispatch 2024-09-21T02:05:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]} : dispatch 2024-09-21T02:05:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]} : dispatch 2024-09-21T02:05:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:05:54.145 INFO:teuthology.orchestra.run.smithi016.stdout:60129542224 2024-09-21T02:05:54.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T02:05:54.575 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-09-21T02:05:54.579 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-09-21T02:05:54.579 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T02:05:54.579 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:54 smithi016 ceph-mon[31245]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:05:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 -- 172.21.15.16:0/4283389526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac1081c0 msgr2=0x7f77ac1085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 --2- 172.21.15.16:0/4283389526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac1081c0 0x7f77ac1085d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f779c009900 tx=0x7f779c02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 -- 172.21.15.16:0/4283389526 shutdown_connections 2024-09-21T02:05:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 --2- 172.21.15.16:0/4283389526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77ac108b10 0x7f77ac10f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 --2- 172.21.15.16:0/4283389526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac1081c0 0x7f77ac1085d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 --2- 172.21.15.16:0/4283389526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77ac102880 0x7f77ac102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 -- 172.21.15.16:0/4283389526 >> 172.21.15.16:0/4283389526 conn(0x7f77ac0fe2d0 msgr2=0x7f77ac100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 -- 172.21.15.16:0/4283389526 shutdown_connections 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.573+0000 7f77b319e640 1 -- 172.21.15.16:0/4283389526 wait complete. 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.574+0000 7f77b319e640 1 Processor -- start 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.574+0000 7f77b319e640 1 -- start start 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.574+0000 7f77b319e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77ac102880 0x7f77ac19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.574+0000 7f77b319e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77ac1081c0 0x7f77ac19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b319e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 0x7f77ac19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b319e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac112650 con 0x7f77ac108b10 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b319e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac1124d0 con 0x7f77ac1081c0 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b319e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac1127d0 con 0x7f77ac102880 2024-09-21T02:05:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 0x7f77ac19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 0x7f77ac19fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35626/0 (socket says 172.21.15.16:35626) 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 -- 172.21.15.16:0/3464416597 learned_addr learned my addr 172.21.15.16:0/3464416597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b199b640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77ac1081c0 0x7f77ac19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b219c640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77ac102880 0x7f77ac19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 -- 172.21.15.16:0/3464416597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77ac102880 msgr2=0x7f77ac19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77ac102880 0x7f77ac19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 -- 172.21.15.16:0/3464416597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77ac1081c0 msgr2=0x7f77ac19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77ac1081c0 0x7f77ac19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac1a01a0 con 0x7f77ac108b10 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.575+0000 7f77b299d640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 0x7f77ac19fa70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f77a400eaa0 tx=0x7f77a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a400cdb0 con 0x7f77ac108b10 2024-09-21T02:05:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ac1a0490 con 0x7f77ac108b10 2024-09-21T02:05:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f77a4005020 con 0x7f77ac108b10 2024-09-21T02:05:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a40116d0 con 0x7f77ac108b10 2024-09-21T02:05:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ac1a6bf0 con 0x7f77ac108b10 2024-09-21T02:05:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.576+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77ac101a10 con 0x7f77ac108b10 2024-09-21T02:05:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.579+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f77a4004a40 con 0x7f77ac108b10 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.579+0000 7f77a37fe640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f77980b65f0 0x7f77980b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.579+0000 7f77b219c640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f77980b65f0 0x7f77980b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.579+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f77a40e7e40 con 0x7f77ac108b10 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77a37fe640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] conn(0x7f77980c1b00 0x7f77980c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77a4014ec0 con 0x7f77980c1b00 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f77a40e8290 con 0x7f77ac108b10 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77b199b640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] conn(0x7f77980c1b00 0x7f77980c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77b219c640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f77980b65f0 0x7f77980b8b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77a8009bd0 tx=0x7f77a8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.580+0000 7f77b199b640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] conn(0x7f77980c1b00 0x7f77980c3ff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.581+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== osd.6 v2:172.21.15.16:6818/87656955 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77a4014ec0 con 0x7f77980c1b00 2024-09-21T02:05:54.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 --> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f77ac103b00 con 0x7f77980c1b00 2024-09-21T02:05:54.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77a37fe640 1 -- 172.21.15.16:0/3464416597 <== osd.6 v2:172.21.15.16:6818/87656955 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f77ac103b00 con 0x7f77980c1b00 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] conn(0x7f77980c1b00 msgr2=0x7f77980c3ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77b319e640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] conn(0x7f77980c1b00 0x7f77980c3ff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f77980b65f0 msgr2=0x7f77980b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.596+0000 7f77b319e640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f77980b65f0 0x7f77980b8b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77a8009bd0 tx=0x7f77a8005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 msgr2=0x7f77ac19fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 --2- 172.21.15.16:0/3464416597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ac108b10 0x7f77ac19fa70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f77a400eaa0 tx=0x7f77a400ef60 comp rx=0 tx=0).stop 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b299d640 1 -- 172.21.15.16:0/3464416597 reap_dead start 2024-09-21T02:05:54.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 shutdown_connections 2024-09-21T02:05:54.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 >> 172.21.15.16:0/3464416597 conn(0x7f77ac0fe2d0 msgr2=0x7f77ac104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:54.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 shutdown_connections 2024-09-21T02:05:54.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:54.597+0000 7f77b319e640 1 -- 172.21.15.16:0/3464416597 wait complete. 2024-09-21T02:05:54.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-09-21T02:05:54.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-09-21T02:05:54.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T02:05:54.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:54 smithi161 ceph-mon[34061]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:05:54.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-09-21T02:05:54.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [5, 8]}]': finished 2024-09-21T02:05:54.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-21T02:05:54.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:54 smithi123 ceph-mon[33453]: osdmap e85: 12 total, 12 up, 12 in 2024-09-21T02:05:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:55 smithi016 ceph-mon[31245]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:05:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:55 smithi016 ceph-mon[31245]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T02:05:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:55 smithi016 ceph-mon[31245]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T02:05:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 -- 172.21.15.16:0/3956647585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 msgr2=0x7f50d810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 --2- 172.21.15.16:0/3956647585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d810f520 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009ca0 tx=0x7f50d002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 -- 172.21.15.16:0/3956647585 shutdown_connections 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 --2- 172.21.15.16:0/3956647585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d810f520 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 --2- 172.21.15.16:0/3956647585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 0x7f50d81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 --2- 172.21.15.16:0/3956647585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 0x7f50d8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 -- 172.21.15.16:0/3956647585 >> 172.21.15.16:0/3956647585 conn(0x7f50d8079730 msgr2=0x7f50d8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 -- 172.21.15.16:0/3956647585 shutdown_connections 2024-09-21T02:05:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.613+0000 7f50df3be640 1 -- 172.21.15.16:0/3956647585 wait complete. 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.614+0000 7f50df3be640 1 Processor -- start 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.614+0000 7f50df3be640 1 -- start start 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.614+0000 7f50df3be640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 0x7f50d819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.614+0000 7f50df3be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 0x7f50d819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50df3be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50df3be640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8112860 con 0x7f50d8108cf0 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50df3be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81126e0 con 0x7f50d81083a0 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50df3be640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50d81129e0 con 0x7f50d8102a60 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d819fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35646/0 (socket says 172.21.15.16:35646) 2024-09-21T02:05:55.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 -- 172.21.15.16:0/1642641324 learned_addr learned my addr 172.21.15.16:0/1642641324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd15a640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 0x7f50d819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dc959640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 0x7f50d819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 -- 172.21.15.16:0/1642641324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 msgr2=0x7f50d819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 0x7f50d819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 -- 172.21.15.16:0/1642641324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 msgr2=0x7f50d819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 0x7f50d819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d81a02d0 con 0x7f50d8108cf0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dc959640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50d81083a0 0x7f50d819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd15a640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50d8102a60 0x7f50d819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50dd95b640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d819fba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50d0005cc0 tx=0x7f50d0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.615+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0032c10 con 0x7f50d8108cf0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.616+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50d0032db0 con 0x7f50d8108cf0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.616+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d81a0560 con 0x7f50d8108cf0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.616+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0030a60 con 0x7f50d8108cf0 2024-09-21T02:05:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.616+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d81a6d20 con 0x7f50d8108cf0 2024-09-21T02:05:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.617+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50d8063b50 con 0x7f50d8108cf0 2024-09-21T02:05:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.618+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f50d0030c00 con 0x7f50d8108cf0 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50ce7fc640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50c40b6670 0x7f50c40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50dd15a640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50c40b6670 0x7f50c40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f50d0108d40 con 0x7f50d8108cf0 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50ce7fc640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] conn(0x7f50c40c1b80 0x7f50c40c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50d00020b0 con 0x7f50c40c1b80 2024-09-21T02:05:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f50d0109190 con 0x7f50d8108cf0 2024-09-21T02:05:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.619+0000 7f50dc959640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] conn(0x7f50c40c1b80 0x7f50c40c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.620+0000 7f50dd15a640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50c40b6670 0x7f50c40b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009fc0 tx=0x7f50d4005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.620+0000 7f50dc959640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] conn(0x7f50c40c1b80 0x7f50c40c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.621+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== osd.9 v2:172.21.15.16:6826/638537789 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50d00020b0 con 0x7f50c40c1b80 2024-09-21T02:05:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.634+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 --> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50d8063f50 con 0x7f50c40c1b80 2024-09-21T02:05:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.634+0000 7f50ce7fc640 1 -- 172.21.15.16:0/1642641324 <== osd.9 v2:172.21.15.16:6826/638537789 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f50d8063f50 con 0x7f50c40c1b80 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] conn(0x7f50c40c1b80 msgr2=0x7f50c40c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] conn(0x7f50c40c1b80 0x7f50c40c4070 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50c40b6670 msgr2=0x7f50c40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f50c40b6670 0x7f50c40b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009fc0 tx=0x7f50d4005f90 comp rx=0 tx=0).stop 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 msgr2=0x7f50d819fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 --2- 172.21.15.16:0/1642641324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d8108cf0 0x7f50d819fba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50d0005cc0 tx=0x7f50d0003f90 comp rx=0 tx=0).stop 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50dd95b640 1 -- 172.21.15.16:0/1642641324 reap_dead start 2024-09-21T02:05:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 shutdown_connections 2024-09-21T02:05:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 >> 172.21.15.16:0/1642641324 conn(0x7f50d8079730 msgr2=0x7f50d806b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 shutdown_connections 2024-09-21T02:05:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:55.635+0000 7f50df3be640 1 -- 172.21.15.16:0/1642641324 wait complete. 2024-09-21T02:05:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:55 smithi161 ceph-mon[34061]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:05:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:55 smithi161 ceph-mon[34061]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T02:05:55.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:55 smithi161 ceph-mon[34061]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T02:05:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:55 smithi123 ceph-mon[33453]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:05:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:55 smithi123 ceph-mon[33453]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T02:05:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:55 smithi123 ceph-mon[33453]: osdmap e86: 12 total, 12 up, 12 in 2024-09-21T02:05:56.133 INFO:teuthology.orchestra.run.smithi016.stdout:231928234028 2024-09-21T02:05:56.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.8 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 -- 172.21.15.16:0/1365633644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc81094f0 msgr2=0x7efcc8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 --2- 172.21.15.16:0/1365633644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc81094f0 0x7efcc8109900 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009900 tx=0x7efcbc02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 -- 172.21.15.16:0/1365633644 shutdown_connections 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 --2- 172.21.15.16:0/1365633644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc8109e40 0x7efcc8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 --2- 172.21.15.16:0/1365633644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc81094f0 0x7efcc8109900 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 --2- 172.21.15.16:0/1365633644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc8103bb0 0x7efcc8103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 -- 172.21.15.16:0/1365633644 >> 172.21.15.16:0/1365633644 conn(0x7efcc80ff600 msgr2=0x7efcc8101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 -- 172.21.15.16:0/1365633644 shutdown_connections 2024-09-21T02:05:56.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.085+0000 7efccd249640 1 -- 172.21.15.16:0/1365633644 wait complete. 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.086+0000 7efccd249640 1 Processor -- start 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.086+0000 7efccd249640 1 -- start start 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 0x7efcc819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc81094f0 0x7efcc819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc8109e40 0x7efcc819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcc81139b0 con 0x7efcc8103bb0 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8113830 con 0x7efcc8109e40 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efccd249640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8113b30 con 0x7efcc81094f0 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 0x7efcc819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 0x7efcc819a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35668/0 (socket says 172.21.15.16:35668) 2024-09-21T02:05:56.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 -- 172.21.15.16:0/734993167 learned_addr learned my addr 172.21.15.16:0/734993167 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc759e640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc8109e40 0x7efcc819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc659c640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc81094f0 0x7efcc819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 -- 172.21.15.16:0/734993167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc81094f0 msgr2=0x7efcc819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc81094f0 0x7efcc819a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 -- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc8109e40 msgr2=0x7efcc819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc8109e40 0x7efcc819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc6d9d640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc81a0350 con 0x7efcc8103bb0 2024-09-21T02:05:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.087+0000 7efcc659c640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcc81094f0 0x7efcc819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:05:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efcc6d9d640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 0x7efcc819a3e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efcb800d950 tx=0x7efcb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8017070 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc81a0640 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efcb8009c30 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb800f620 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.088+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc81a6da0 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.089+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efcc8063b50 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.091+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7efcb8010460 con 0x7efcc8103bb0 2024-09-21T02:05:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.091+0000 7efcaffff640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7efcb00b65f0 0x7efcb00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.091+0000 7efcc659c640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7efcb00b65f0 0x7efcb00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.091+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7efcb80e7b40 con 0x7efcc8103bb0 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.092+0000 7efcaffff640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] conn(0x7efcb00c1b00 0x7efcb00c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.092+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcb00c46c0 con 0x7efcb00c1b00 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.092+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7efcb80e7f90 con 0x7efcc8103bb0 2024-09-21T02:05:56.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.092+0000 7efcc659c640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7efcb00b65f0 0x7efcb00b8b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0057c0 tx=0x7efcbc009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.092+0000 7efcc759e640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] conn(0x7efcb00c1b00 0x7efcb00c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.093+0000 7efcc759e640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] conn(0x7efcb00c1b00 0x7efcb00c3ff0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.093+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== osd.4 v2:172.21.15.123:6808/1713366757 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efcb00c46c0 con 0x7efcb00c1b00 2024-09-21T02:05:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.108+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 --> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efcc8063f80 con 0x7efcb00c1b00 2024-09-21T02:05:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.109+0000 7efcaffff640 1 -- 172.21.15.16:0/734993167 <== osd.4 v2:172.21.15.123:6808/1713366757 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efcc8063f80 con 0x7efcb00c1b00 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] conn(0x7efcb00c1b00 msgr2=0x7efcb00c3ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] conn(0x7efcb00c1b00 0x7efcb00c3ff0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7efcb00b65f0 msgr2=0x7efcb00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7efcb00b65f0 0x7efcb00b8b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0057c0 tx=0x7efcbc009680 comp rx=0 tx=0).stop 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 msgr2=0x7efcc819a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 --2- 172.21.15.16:0/734993167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc8103bb0 0x7efcc819a3e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efcb800d950 tx=0x7efcb800de10 comp rx=0 tx=0).stop 2024-09-21T02:05:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efcc759e640 1 -- 172.21.15.16:0/734993167 reap_dead start 2024-09-21T02:05:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 shutdown_connections 2024-09-21T02:05:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 >> 172.21.15.16:0/734993167 conn(0x7efcc80ff600 msgr2=0x7efcc8101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 shutdown_connections 2024-09-21T02:05:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.110+0000 7efccd249640 1 -- 172.21.15.16:0/734993167 wait complete. 2024-09-21T02:05:56.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.866+0000 7f2901c18640 1 -- 172.21.15.16:0/1353219118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc104900 msgr2=0x7f28fc104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.866+0000 7f2901c18640 1 --2- 172.21.15.16:0/1353219118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc104900 0x7f28fc104d10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009900 tx=0x7f28ec02a410 comp rx=0 tx=0).stop 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 -- 172.21.15.16:0/1353219118 shutdown_connections 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 --2- 172.21.15.16:0/1353219118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc10deb0 0x7f28fc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 --2- 172.21.15.16:0/1353219118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28fc105250 0x7f28fc10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 --2- 172.21.15.16:0/1353219118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc104900 0x7f28fc104d10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 -- 172.21.15.16:0/1353219118 >> 172.21.15.16:0/1353219118 conn(0x7f28fc079760 msgr2=0x7f28fc079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 -- 172.21.15.16:0/1353219118 shutdown_connections 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 -- 172.21.15.16:0/1353219118 wait complete. 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 Processor -- start 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.867+0000 7f2901c18640 1 -- start start 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc104900 0x7f28fc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 0x7f28fc19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28fc10deb0 0x7f28fc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc113970 con 0x7f28fc105250 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1137f0 con 0x7f28fc104900 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f2901c18640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc113af0 con 0x7f28fc10deb0 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 0x7f28fc19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 0x7f28fc19a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35684/0 (socket says 172.21.15.16:35684) 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28fb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc104900 0x7f28fc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 -- 172.21.15.16:0/1935448354 learned_addr learned my addr 172.21.15.16:0/1935448354 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28fbfff640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28fc10deb0 0x7f28fc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 -- 172.21.15.16:0/1935448354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28fc10deb0 msgr2=0x7f28fc19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28fc10deb0 0x7f28fc19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 -- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc104900 msgr2=0x7f28fc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc104900 0x7f28fc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28faffd640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc1a0350 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.868+0000 7f28fb7fe640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28fc104900 0x7f28fc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f28faffd640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 0x7f28fc19a980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f28f000e9f0 tx=0x7f28f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f000ccc0 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f28f00054b0 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0010430 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc1a0640 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.869+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1a6da0 con 0x7f28fc105250 2024-09-21T02:05:56.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.870+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28fc063b50 con 0x7f28fc105250 2024-09-21T02:05:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.872+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f28f00046c0 con 0x7f28fc105250 2024-09-21T02:05:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28f8ff9640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f28e40b6670 0x7f28e40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28fb7fe640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f28e40b6670 0x7f28e40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f28f00e82a0 con 0x7f28fc105250 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28f8ff9640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] conn(0x7f28e40c1b80 0x7f28e40c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28f0011e50 con 0x7f28e40c1b80 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.873+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f28f00e86f0 con 0x7f28fc105250 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.874+0000 7f28fb7fe640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f28e40b6670 0x7f28e40b8b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004770 tx=0x7f28ec0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.874+0000 7f28fbfff640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] conn(0x7f28e40c1b80 0x7f28e40c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.874+0000 7f28fbfff640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] conn(0x7f28e40c1b80 0x7f28e40c4070 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.875+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== osd.10 v2:172.21.15.123:6824/1402496387 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28f0011e50 con 0x7f28e40c1b80 2024-09-21T02:05:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.888+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 --> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f28fc063f50 con 0x7f28e40c1b80 2024-09-21T02:05:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.889+0000 7f28f8ff9640 1 -- 172.21.15.16:0/1935448354 <== osd.10 v2:172.21.15.123:6824/1402496387 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f28fc063f50 con 0x7f28e40c1b80 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.889+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] conn(0x7f28e40c1b80 msgr2=0x7f28e40c4070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.889+0000 7f2901c18640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] conn(0x7f28e40c1b80 0x7f28e40c4070 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f28e40b6670 msgr2=0x7f28e40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f28e40b6670 0x7f28e40b8b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004770 tx=0x7f28ec0058d0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 msgr2=0x7f28fc19a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 --2- 172.21.15.16:0/1935448354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc105250 0x7f28fc19a980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f28f000e9f0 tx=0x7f28f000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f28fbfff640 1 -- 172.21.15.16:0/1935448354 reap_dead start 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 shutdown_connections 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 >> 172.21.15.16:0/1935448354 conn(0x7f28fc079760 msgr2=0x7f28fc0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 shutdown_connections 2024-09-21T02:05:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:56.890+0000 7f2901c18640 1 -- 172.21.15.16:0/1935448354 wait complete. 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- 172.21.15.16:0/1129966082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102fa0 msgr2=0x7f6fc010fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 --2- 172.21.15.16:0/1129966082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102fa0 0x7f6fc010fcc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009900 tx=0x7f6fb002a480 comp rx=0 tx=0).stop 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- 172.21.15.16:0/1129966082 shutdown_connections 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 --2- 172.21.15.16:0/1129966082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0110200 0x7f6fc0112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 --2- 172.21.15.16:0/1129966082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102fa0 0x7f6fc010fcc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 --2- 172.21.15.16:0/1129966082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc0102650 0x7f6fc0102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- 172.21.15.16:0/1129966082 >> 172.21.15.16:0/1129966082 conn(0x7f6fc0079760 msgr2=0x7f6fc0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- 172.21.15.16:0/1129966082 shutdown_connections 2024-09-21T02:05:57.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- 172.21.15.16:0/1129966082 wait complete. 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 Processor -- start 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.302+0000 7f6fc772a640 1 -- start start 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 0x7f6fc019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0102fa0 0x7f6fc019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc0110200 0x7f6fc01a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0115b50 con 0x7f6fc0102650 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01159d0 con 0x7f6fc0110200 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc772a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0115cd0 con 0x7f6fc0102fa0 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 0x7f6fc019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 0x7f6fc019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42952/0 (socket says 172.21.15.16:42952) 2024-09-21T02:05:57.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 -- 172.21.15.16:0/456891077 learned_addr learned my addr 172.21.15.16:0/456891077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc5cc7640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc0110200 0x7f6fc01a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc4cc5640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0102fa0 0x7f6fc019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 -- 172.21.15.16:0/456891077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0102fa0 msgr2=0x7f6fc019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0102fa0 0x7f6fc019cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 -- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc0110200 msgr2=0x7f6fc01a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fc0110200 0x7f6fc01a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc01a24d0 con 0x7f6fc0102650 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc4cc5640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fc0102fa0 0x7f6fc019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.303+0000 7f6fc54c6640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 0x7f6fc019c560 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d950 tx=0x7f6fbc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.304+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017070 con 0x7f6fc0102650 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.304+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc01a27c0 con 0x7f6fc0102650 2024-09-21T02:05:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.304+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6fbc009c30 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.304+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00f620 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.304+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc01a8f20 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.305+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6fc0063b50 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.306+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6fbc010460 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fb67fc640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6fac0b65f0 0x7f6fac0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fc4cc5640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6fac0b65f0 0x7f6fac0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f6fbc0e79b0 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fb67fc640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] conn(0x7f6fac0c1b00 0x7f6fac0c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fac0c46c0 con 0x7f6fac0c1b00 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.307+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6fbc0e7e00 con 0x7f6fc0102650 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.308+0000 7f6fc4cc5640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6fac0b65f0 0x7f6fac0b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0006010 tx=0x7f6fb0000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.308+0000 7f6fc5cc7640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] conn(0x7f6fac0c1b00 0x7f6fac0c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.308+0000 7f6fc5cc7640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] conn(0x7f6fac0c1b00 0x7f6fac0c3ff0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.309+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== osd.7 v2:172.21.15.123:6816/271400775 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6fac0c46c0 con 0x7f6fac0c1b00 2024-09-21T02:05:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.323+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 --> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6fc0063f50 con 0x7f6fac0c1b00 2024-09-21T02:05:57.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.324+0000 7f6fb67fc640 1 -- 172.21.15.16:0/456891077 <== osd.7 v2:172.21.15.123:6816/271400775 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6fc0063f50 con 0x7f6fac0c1b00 2024-09-21T02:05:57.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] conn(0x7f6fac0c1b00 msgr2=0x7f6fac0c3ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] conn(0x7f6fac0c1b00 0x7f6fac0c3ff0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6fac0b65f0 msgr2=0x7f6fac0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6fac0b65f0 0x7f6fac0b8b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0006010 tx=0x7f6fb0000bc0 comp rx=0 tx=0).stop 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 msgr2=0x7f6fc019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 --2- 172.21.15.16:0/456891077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fc0102650 0x7f6fc019c560 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d950 tx=0x7f6fbc00de10 comp rx=0 tx=0).stop 2024-09-21T02:05:57.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc5cc7640 1 -- 172.21.15.16:0/456891077 reap_dead start 2024-09-21T02:05:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 shutdown_connections 2024-09-21T02:05:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 >> 172.21.15.16:0/456891077 conn(0x7f6fc0079760 msgr2=0x7f6fc01116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 shutdown_connections 2024-09-21T02:05:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:57.325+0000 7f6fc772a640 1 -- 172.21.15.16:0/456891077 wait complete. 2024-09-21T02:05:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:57 smithi016 ceph-mon[31245]: pgmap v316: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-09-21T02:05:57.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:57 smithi161 ceph-mon[34061]: pgmap v316: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-09-21T02:05:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:57 smithi123 ceph-mon[33453]: pgmap v316: 129 pgs: 3 peering, 126 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-09-21T02:05:58.067 INFO:teuthology.orchestra.run.smithi016.stdout:279172874274 2024-09-21T02:05:58.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.10 2024-09-21T02:05:58.740 INFO:teuthology.orchestra.run.smithi016.stdout:210453397554 2024-09-21T02:05:58.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T02:05:59.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 -- 172.21.15.16:0/3564281764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc109df0 msgr2=0x7f65cc110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 --2- 172.21.15.16:0/3564281764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc109df0 0x7f65cc110620 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009ca0 tx=0x7f65c402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 -- 172.21.15.16:0/3564281764 shutdown_connections 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 --2- 172.21.15.16:0/3564281764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc109df0 0x7f65cc110620 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 --2- 172.21.15.16:0/3564281764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc1094a0 0x7f65cc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 --2- 172.21.15.16:0/3564281764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65cc103b60 0x7f65cc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 -- 172.21.15.16:0/3564281764 >> 172.21.15.16:0/3564281764 conn(0x7f65cc07b850 msgr2=0x7f65cc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 -- 172.21.15.16:0/3564281764 shutdown_connections 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.173+0000 7f65d3527640 1 -- 172.21.15.16:0/3564281764 wait complete. 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.174+0000 7f65d3527640 1 Processor -- start 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.174+0000 7f65d3527640 1 -- start start 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc103b60 0x7f65cc19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 0x7f65cc19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65cc109df0 0x7f65cc19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc1139b0 con 0x7f65cc1094a0 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc113830 con 0x7f65cc103b60 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc113b30 con 0x7f65cc109df0 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 0x7f65cc19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 0x7f65cc19aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42974/0 (socket says 172.21.15.16:42974) 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 -- 172.21.15.16:0/2505150470 learned_addr learned my addr 172.21.15.16:0/2505150470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:05:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d12c3640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc103b60 0x7f65cc19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d1ac4640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65cc109df0 0x7f65cc19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 -- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65cc109df0 msgr2=0x7f65cc19fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65cc109df0 0x7f65cc19fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 -- 172.21.15.16:0/2505150470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc103b60 msgr2=0x7f65cc19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc103b60 0x7f65cc19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc1a0370 con 0x7f65cc1094a0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d12c3640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65cc103b60 0x7f65cc19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d0ac2640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 0x7f65cc19aa60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c8c0 tx=0x7f65bc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc007bb0 con 0x7f65cc1094a0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65bc007d50 con 0x7f65cc1094a0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0186a0 con 0x7f65cc1094a0 2024-09-21T02:05:59.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.175+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65cc1a0660 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.176+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65cc1a6e80 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.176+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65cc063ad0 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.178+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f65bc00fa00 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65c27fc640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f65b80b66a0 0x7f65b80b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65d12c3640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f65b80b66a0 0x7f65b80b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f65bc0e7eb0 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65c27fc640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] conn(0x7f65b80c1bb0 0x7f65b80c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65bc01ee50 con 0x7f65b80c1bb0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f65bc0e8260 con 0x7f65cc1094a0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65d12c3640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f65b80b66a0 0x7f65b80b8bb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009b40 tx=0x7f65c8005e50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.179+0000 7f65d1ac4640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] conn(0x7f65b80c1bb0 0x7f65b80c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.180+0000 7f65d1ac4640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] conn(0x7f65b80c1bb0 0x7f65b80c40a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:05:59.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.181+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== osd.11 v2:172.21.15.161:6824/1648324302 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65bc01ee50 con 0x7f65b80c1bb0 2024-09-21T02:05:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.195+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 --> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65cc063ed0 con 0x7f65b80c1bb0 2024-09-21T02:05:59.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.196+0000 7f65c27fc640 1 -- 172.21.15.16:0/2505150470 <== osd.11 v2:172.21.15.161:6824/1648324302 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f65cc063ed0 con 0x7f65b80c1bb0 2024-09-21T02:05:59.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.196+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] conn(0x7f65b80c1bb0 msgr2=0x7f65b80c40a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.196+0000 7f65d3527640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] conn(0x7f65b80c1bb0 0x7f65b80c40a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f65b80b66a0 msgr2=0x7f65b80b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f65b80b66a0 0x7f65b80b8bb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009b40 tx=0x7f65c8005e50 comp rx=0 tx=0).stop 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 msgr2=0x7f65cc19aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 --2- 172.21.15.16:0/2505150470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65cc1094a0 0x7f65cc19aa60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c8c0 tx=0x7f65bc00cd80 comp rx=0 tx=0).stop 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d1ac4640 1 -- 172.21.15.16:0/2505150470 reap_dead start 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 shutdown_connections 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 >> 172.21.15.16:0/2505150470 conn(0x7f65cc07b850 msgr2=0x7f65cc105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 shutdown_connections 2024-09-21T02:05:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:05:59.197+0000 7f65d3527640 1 -- 172.21.15.16:0/2505150470 wait complete. 2024-09-21T02:05:59.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:05:59 smithi016 ceph-mon[31245]: pgmap v317: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-09-21T02:05:59.655 INFO:teuthology.orchestra.run.smithi016.stdout:188978561079 2024-09-21T02:05:59.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T02:05:59.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:05:59 smithi161 ceph-mon[34061]: pgmap v317: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-09-21T02:05:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:05:59 smithi123 ceph-mon[33453]: pgmap v317: 129 pgs: 1 peering, 128 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-09-21T02:06:00.091 INFO:teuthology.orchestra.run.smithi016.stdout:141733920833 2024-09-21T02:06:00.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T02:06:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:00 smithi016 ceph-mon[31245]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T02:06:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:00 smithi016 ceph-mon[31245]: Cluster is now healthy 2024-09-21T02:06:00.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:00 smithi161 ceph-mon[34061]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T02:06:00.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:00 smithi161 ceph-mon[34061]: Cluster is now healthy 2024-09-21T02:06:00.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:00 smithi123 ceph-mon[33453]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T02:06:00.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:00 smithi123 ceph-mon[33453]: Cluster is now healthy 2024-09-21T02:06:00.909 INFO:teuthology.orchestra.run.smithi016.stdout:257698037800 2024-09-21T02:06:00.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.9 2024-09-21T02:06:01.325 INFO:teuthology.orchestra.run.smithi016.stdout:300647710751 2024-09-21T02:06:01.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.11 2024-09-21T02:06:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:01.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:01 smithi016 ceph-mon[31245]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-21T02:06:01.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:01 smithi161 ceph-mon[34061]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-21T02:06:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:01 smithi123 ceph-mon[33453]: pgmap v318: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-09-21T02:06:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 -- 172.21.15.16:0/1859259832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b810de10 msgr2=0x7f22b8110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 --2- 172.21.15.16:0/1859259832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b810de10 0x7f22b8110300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009900 tx=0x7f22b002a410 comp rx=0 tx=0).stop 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 -- 172.21.15.16:0/1859259832 shutdown_connections 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 --2- 172.21.15.16:0/1859259832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b810de10 0x7f22b8110300 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 --2- 172.21.15.16:0/1859259832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b8105300 0x7f22b810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 --2- 172.21.15.16:0/1859259832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b81049b0 0x7f22b8104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 -- 172.21.15.16:0/1859259832 >> 172.21.15.16:0/1859259832 conn(0x7f22b8079730 msgr2=0x7f22b8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 -- 172.21.15.16:0/1859259832 shutdown_connections 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.265+0000 7f22bfb87640 1 -- 172.21.15.16:0/1859259832 wait complete. 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 Processor -- start 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 -- start start 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b81049b0 0x7f22b819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b810de10 0x7f22b819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8113940 con 0x7f22b8105300 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81137c0 con 0x7f22b81049b0 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bfb87640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8113ac0 con 0x7f22b810de10 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bd122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bd122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42988/0 (socket says 172.21.15.16:42988) 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.266+0000 7f22bd122640 1 -- 172.21.15.16:0/3917372758 learned_addr learned my addr 172.21.15.16:0/3917372758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd923640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b81049b0 0x7f22b819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22be124640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b810de10 0x7f22b819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 -- 172.21.15.16:0/3917372758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b810de10 msgr2=0x7f22b819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b810de10 0x7f22b819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 -- 172.21.15.16:0/3917372758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b81049b0 msgr2=0x7f22b819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b81049b0 0x7f22b819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 -- 172.21.15.16:0/3917372758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b81a0360 con 0x7f22b8105300 2024-09-21T02:06:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bd122640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00d740 tx=0x7f22ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac017070 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f22ac00de50 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b81a0650 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac00f620 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.267+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b81a6db0 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.268+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b8104dc0 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.270+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f22ac01a070 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.271+0000 7f22aaffd640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 0x7f22a40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.271+0000 7f22bd923640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 0x7f22a40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.271+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f22ac0e7300 con 0x7f22b8105300 2024-09-21T02:06:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.272+0000 7f22bd923640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 0x7f22a40b8b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005c40 tx=0x7f22b400a590 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.274+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f22ac0ad5a0 con 0x7f22b8105300 2024-09-21T02:06:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.466+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f22b81a70b0 con 0x7f22b8105300 2024-09-21T02:06:02.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.466+0000 7f22aaffd640 1 -- 172.21.15.16:0/3917372758 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f22ac0b1980 con 0x7f22b8105300 2024-09-21T02:06:02.468 INFO:teuthology.orchestra.run.smithi016.stdout:60129542227 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 msgr2=0x7f22a40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 0x7f22a40b8b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005c40 tx=0x7f22b400a590 comp rx=0 tx=0).stop 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 msgr2=0x7f22b819a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00d740 tx=0x7f22ac00dc00 comp rx=0 tx=0).stop 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 shutdown_connections 2024-09-21T02:06:02.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f22a40b65f0 0x7f22a40b8b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22b810de10 0x7f22b819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b8105300 0x7f22b819a990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 --2- 172.21.15.16:0/3917372758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22b81049b0 0x7f22b819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 >> 172.21.15.16:0/3917372758 conn(0x7f22b8079730 msgr2=0x7f22b80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 shutdown_connections 2024-09-21T02:06:02.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.469+0000 7f22bfb87640 1 -- 172.21.15.16:0/3917372758 wait complete. 2024-09-21T02:06:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878117455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 msgr2=0x7f9ef810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878117455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef810f520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009ca0 tx=0x7f9ef002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878117455 shutdown_connections 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878117455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef810f520 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878117455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef81083a0 0x7f9ef81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878117455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef8102a60 0x7f9ef8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878117455 >> 172.21.15.16:0/3878117455 conn(0x7f9ef8079730 msgr2=0x7f9ef8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878117455 shutdown_connections 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.501+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878117455 wait complete. 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.502+0000 7f9efe66c640 1 Processor -- start 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.502+0000 7f9efe66c640 1 -- start start 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef8102a60 0x7f9ef819b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef81083a0 0x7f9ef819b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8112860 con 0x7f9ef8108cf0 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81126e0 con 0x7f9ef81083a0 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efe66c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81129e0 con 0x7f9ef8102a60 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43008/0 (socket says 172.21.15.16:43008) 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 -- 172.21.15.16:0/3878827953 learned_addr learned my addr 172.21.15.16:0/3878827953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9ef77fe640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef81083a0 0x7f9ef819b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef8102a60 0x7f9ef819b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 -- 172.21.15.16:0/3878827953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef8102a60 msgr2=0x7f9ef819b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef8102a60 0x7f9ef819b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 -- 172.21.15.16:0/3878827953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef81083a0 msgr2=0x7f9ef819b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef81083a0 0x7f9ef819b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.503+0000 7f9efcc09640 1 -- 172.21.15.16:0/3878827953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef81a63a0 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9efcc09640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005cc0 tx=0x7f9ef0032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0032e00 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9ef00044d0 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef81a6630 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0030b70 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.504+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef81a6b70 con 0x7f9ef8108cf0 2024-09-21T02:06:02.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.505+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef8102e70 con 0x7f9ef8108cf0 2024-09-21T02:06:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.507+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f9ef0030d10 con 0x7f9ef8108cf0 2024-09-21T02:06:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.508+0000 7f9ef57fa640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 0x7f9ee00b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.508+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 0x7f9ee00b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.508+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f9ef0108da0 con 0x7f9ef8108cf0 2024-09-21T02:06:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.509+0000 7f9ef7fff640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 0x7f9ee00b8b30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9ef806b9c0 tx=0x7f9ee800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.511+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9ef00cefe0 con 0x7f9ef8108cf0 2024-09-21T02:06:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.703+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9ef802e640 con 0x7f9ef8108cf0 2024-09-21T02:06:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.703+0000 7f9ef57fa640 1 -- 172.21.15.16:0/3878827953 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9ef00d33c0 con 0x7f9ef8108cf0 2024-09-21T02:06:02.704 INFO:teuthology.orchestra.run.smithi016.stdout:115964117064 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 msgr2=0x7f9ee00b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 0x7f9ee00b8b30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9ef806b9c0 tx=0x7f9ee800b390 comp rx=0 tx=0).stop 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 msgr2=0x7f9ef819bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005cc0 tx=0x7f9ef0032a10 comp rx=0 tx=0).stop 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 shutdown_connections 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9ee00b6620 0x7f9ee00b8b30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ef8108cf0 0x7f9ef819bb90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ef81083a0 0x7f9ef819b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 --2- 172.21.15.16:0/3878827953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef8102a60 0x7f9ef819b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:02.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 >> 172.21.15.16:0/3878827953 conn(0x7f9ef8079730 msgr2=0x7f9ef81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:02.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 shutdown_connections 2024-09-21T02:06:02.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:02.706+0000 7f9efe66c640 1 -- 172.21.15.16:0/3878827953 wait complete. 2024-09-21T02:06:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:03.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117062 got 115964117064 for osd.3 2024-09-21T02:06:03.417 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:03 smithi016 ceph-mon[31245]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T02:06:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3917372758' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:06:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3878827953' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:06:03.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:03 smithi161 ceph-mon[34061]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T02:06:03.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3917372758' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:06:03.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3878827953' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:06:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:03 smithi123 ceph-mon[33453]: pgmap v319: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-21T02:06:03.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3917372758' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T02:06:03.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3878827953' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T02:06:04.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:04 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2132119305' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:06:04.615 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542224 got 60129542227 for osd.1 2024-09-21T02:06:04.615 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:04.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:04 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2132119305' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:06:04.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:04 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2132119305' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-21T02:06:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.805+0000 7f6679bfa640 1 -- 172.21.15.16:0/1702514567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6674102a80 msgr2=0x7f6674102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.805+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1702514567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6674102a80 0x7f6674102e90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6664009900 tx=0x7f666402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 -- 172.21.15.16:0/1702514567 shutdown_connections 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1702514567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674108d10 0x7f667410f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1702514567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66741083c0 0x7f66741087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1702514567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6674102a80 0x7f6674102e90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 -- 172.21.15.16:0/1702514567 >> 172.21.15.16:0/1702514567 conn(0x7f6674079740 msgr2=0x7f6674079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 -- 172.21.15.16:0/1702514567 shutdown_connections 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.806+0000 7f6679bfa640 1 -- 172.21.15.16:0/1702514567 wait complete. 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 Processor -- start 2024-09-21T02:06:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 -- start start 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 0x7f6674199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6674108d10 0x7f667419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66741128b0 con 0x7f66741083c0 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6674112730 con 0x7f6674102a80 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6679bfa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6674112a30 con 0x7f6674108d10 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43034/0 (socket says 172.21.15.16:43034) 2024-09-21T02:06:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6672ffd640 1 -- 172.21.15.16:0/1445163126 learned_addr learned my addr 172.21.15.16:0/1445163126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f66737fe640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 0x7f6674199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.807+0000 7f6673fff640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6674108d10 0x7f667419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 -- 172.21.15.16:0/1445163126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6674108d10 msgr2=0x7f667419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6674108d10 0x7f667419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 -- 172.21.15.16:0/1445163126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 msgr2=0x7f6674199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 0x7f6674199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 -- 172.21.15.16:0/1445163126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667419f290 con 0x7f66741083c0 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f66737fe640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 0x7f6674199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6672ffd640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f666800d740 tx=0x7f666800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:04.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668017070 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f666800de50 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668004c60 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667419f580 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.808+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66741a5ce0 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.809+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6674103d00 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.811+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6668004e00 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.812+0000 7f6670ff9640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 0x7f665c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.812+0000 7f66737fe640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 0x7f665c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.812+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f66680e7800 con 0x7f66741083c0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.813+0000 7f66737fe640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 0x7f665c0b8b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6664000c00 tx=0x7f6664005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.815+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f66680adaa0 con 0x7f66741083c0 2024-09-21T02:06:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.007+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6674102e90 con 0x7f66741083c0 2024-09-21T02:06:05.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.008+0000 7f6670ff9640 1 -- 172.21.15.16:0/1445163126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f66680b1e80 con 0x7f66741083c0 2024-09-21T02:06:05.009 INFO:teuthology.orchestra.run.smithi016.stdout:163208757311 2024-09-21T02:06:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 msgr2=0x7f665c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 0x7f665c0b8b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6664000c00 tx=0x7f6664005a20 comp rx=0 tx=0).stop 2024-09-21T02:06:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 msgr2=0x7f66741998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f666800d740 tx=0x7f666800dc00 comp rx=0 tx=0).stop 2024-09-21T02:06:05.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 shutdown_connections 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f665c0b65f0 0x7f665c0b8b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6674108d10 0x7f667419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66741083c0 0x7f66741998c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 --2- 172.21.15.16:0/1445163126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6674102a80 0x7f6674199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 >> 172.21.15.16:0/1445163126 conn(0x7f6674079740 msgr2=0x7f6674101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.010+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 shutdown_connections 2024-09-21T02:06:05.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.011+0000 7f6679bfa640 1 -- 172.21.15.16:0/1445163126 wait complete. 2024-09-21T02:06:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:05 smithi016 ceph-mon[31245]: pgmap v320: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:06:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:05 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1445163126' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:06:05.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:05 smithi161 ceph-mon[34061]: pgmap v320: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:06:05.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:05 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1445163126' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:06:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:05 smithi123 ceph-mon[33453]: pgmap v320: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:06:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:05 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1445163126' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T02:06:05.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757307 got 163208757311 for osd.5 2024-09-21T02:06:05.780 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 -- 172.21.15.16:0/3195283614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4109df0 msgr2=0x7f0ef4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 --2- 172.21.15.16:0/3195283614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4109df0 0x7f0ef4110620 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009ca0 tx=0x7f0eec02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 -- 172.21.15.16:0/3195283614 shutdown_connections 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 --2- 172.21.15.16:0/3195283614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4109df0 0x7f0ef4110620 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 --2- 172.21.15.16:0/3195283614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 0x7f0ef41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 --2- 172.21.15.16:0/3195283614 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4103b60 0x7f0ef4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 -- 172.21.15.16:0/3195283614 >> 172.21.15.16:0/3195283614 conn(0x7f0ef407b850 msgr2=0x7f0ef407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 -- 172.21.15.16:0/3195283614 shutdown_connections 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 -- 172.21.15.16:0/3195283614 wait complete. 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.967+0000 7f0efa784640 1 Processor -- start 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 -- start start 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 0x7f0ef419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 0x7f0ef419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41139b0 con 0x7f0ef4103b60 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4113830 con 0x7f0ef41094a0 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0efa784640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4113b30 con 0x7f0ef4109df0 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43062/0 (socket says 172.21.15.16:43062) 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 -- 172.21.15.16:0/2897646165 learned_addr learned my addr 172.21.15.16:0/2897646165 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef37fe640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 0x7f0ef419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef8d21640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 0x7f0ef419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 -- 172.21.15.16:0/2897646165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 msgr2=0x7f0ef419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 0x7f0ef419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 -- 172.21.15.16:0/2897646165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 msgr2=0x7f0ef419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 0x7f0ef419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef3fff640 1 -- 172.21.15.16:0/2897646165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef41a0370 con 0x7f0ef4103b60 2024-09-21T02:06:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.968+0000 7f0ef8d21640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 0x7f0ef419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0ef3fff640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400d770 tx=0x7f0ee400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4017070 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ee40054b0 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41a0660 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004c60 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.969+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41a6e80 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.970+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef4104de0 con 0x7f0ef4103b60 2024-09-21T02:06:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.972+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0ee4004e00 con 0x7f0ef4103b60 2024-09-21T02:06:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.973+0000 7f0ef17fa640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 0x7f0edc0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.973+0000 7f0ef37fe640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 0x7f0edc0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.973+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0ee40e78f0 con 0x7f0ef4103b60 2024-09-21T02:06:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.973+0000 7f0ef37fe640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 0x7f0edc0b8b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ef41071a0 tx=0x7f0ee800c4a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:05.976+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0ee40adb30 con 0x7f0ef4103b60 2024-09-21T02:06:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.169+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0ef410bbe0 con 0x7f0ef4103b60 2024-09-21T02:06:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.170+0000 7f0ef17fa640 1 -- 172.21.15.16:0/2897646165 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ee40b1f10 con 0x7f0ef4103b60 2024-09-21T02:06:06.171 INFO:teuthology.orchestra.run.smithi016.stdout:38654705752 2024-09-21T02:06:06.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 msgr2=0x7f0edc0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 0x7f0edc0b8b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ef41071a0 tx=0x7f0ee800c4a0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 msgr2=0x7f0ef419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400d770 tx=0x7f0ee400dc30 comp rx=0 tx=0).stop 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 shutdown_connections 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0edc0b6620 0x7f0edc0b8b30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ef4109df0 0x7f0ef419fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ef41094a0 0x7f0ef419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 --2- 172.21.15.16:0/2897646165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ef4103b60 0x7f0ef419a4c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 >> 172.21.15.16:0/2897646165 conn(0x7f0ef407b850 msgr2=0x7f0ef4105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 shutdown_connections 2024-09-21T02:06:06.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.172+0000 7f0efa784640 1 -- 172.21.15.16:0/2897646165 wait complete. 2024-09-21T02:06:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 -- 172.21.15.16:0/712188581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf80108cf0 msgr2=0x7fdf8010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 --2- 172.21.15.16:0/712188581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf80108cf0 0x7fdf8010f520 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009ca0 tx=0x7fdf7802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 -- 172.21.15.16:0/712188581 shutdown_connections 2024-09-21T02:06:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 --2- 172.21.15.16:0/712188581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf80108cf0 0x7fdf8010f520 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 --2- 172.21.15.16:0/712188581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf801083a0 0x7fdf801087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 --2- 172.21.15.16:0/712188581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80102a60 0x7fdf80102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.184+0000 7fdf86e30640 1 -- 172.21.15.16:0/712188581 >> 172.21.15.16:0/712188581 conn(0x7fdf80079730 msgr2=0x7fdf80079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.185+0000 7fdf86e30640 1 -- 172.21.15.16:0/712188581 shutdown_connections 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.185+0000 7fdf86e30640 1 -- 172.21.15.16:0/712188581 wait complete. 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.185+0000 7fdf86e30640 1 Processor -- start 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.185+0000 7fdf86e30640 1 -- start start 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 0x7fdf8019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 0x7fdf8019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80113960 con 0x7fdf801083a0 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801137e0 con 0x7fdf80102a60 2024-09-21T02:06:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf86e30640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80113ae0 con 0x7fdf80108cf0 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43082/0 (socket says 172.21.15.16:43082) 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 -- 172.21.15.16:0/879782441 learned_addr learned my addr 172.21.15.16:0/879782441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf84bcc640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 0x7fdf8019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf853cd640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 0x7fdf8019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 -- 172.21.15.16:0/879782441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 msgr2=0x7fdf8019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 0x7fdf8019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 -- 172.21.15.16:0/879782441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 msgr2=0x7fdf8019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 0x7fdf8019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 -- 172.21.15.16:0/879782441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801a0200 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf853cd640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 0x7fdf8019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf84bcc640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 0x7fdf8019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.186+0000 7fdf7ffff640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c970 tx=0x7fdf7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.187+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74014070 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.187+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf74004440 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.187+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf801a04f0 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.187+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003d40 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.187+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801a6d10 con 0x7fdf801083a0 2024-09-21T02:06:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.188+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf80102e70 con 0x7fdf801083a0 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.190+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fdf74017070 con 0x7fdf801083a0 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.191+0000 7fdf7dffb640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 0x7fdf680b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.191+0000 7fdf84bcc640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 0x7fdf680b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.191+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fdf740e73e0 con 0x7fdf801083a0 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.191+0000 7fdf84bcc640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 0x7fdf680b8bb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf700058c0 tx=0x7fdf7000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.199+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdf7401e030 con 0x7fdf801083a0 2024-09-21T02:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.390+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdf8002e640 con 0x7fdf801083a0 2024-09-21T02:06:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.390+0000 7fdf7dffb640 1 -- 172.21.15.16:0/879782441 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdf740ad680 con 0x7fdf801083a0 2024-09-21T02:06:06.391 INFO:teuthology.orchestra.run.smithi016.stdout:81604378702 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 msgr2=0x7fdf680b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 0x7fdf680b8bb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf700058c0 tx=0x7fdf7000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 msgr2=0x7fdf8019a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c970 tx=0x7fdf7400ce30 comp rx=0 tx=0).stop 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 shutdown_connections 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdf680b66a0 0x7fdf680b8bb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf80108cf0 0x7fdf8019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf801083a0 0x7fdf8019a8f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 --2- 172.21.15.16:0/879782441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf80102a60 0x7fdf8019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 >> 172.21.15.16:0/879782441 conn(0x7fdf80079730 msgr2=0x7fdf80104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 shutdown_connections 2024-09-21T02:06:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:06.393+0000 7fdf86e30640 1 -- 172.21.15.16:0/879782441 wait complete. 2024-09-21T02:06:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:06 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2897646165' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:06:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:06 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/879782441' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:06:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:06 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2897646165' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:06:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:06 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/879782441' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:06:07.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:06 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2897646165' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T02:06:07.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:06 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/879782441' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T02:06:07.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378699 got 81604378702 for osd.2 2024-09-21T02:06:07.205 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 -- 172.21.15.16:0/1508618239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 msgr2=0x7f80c4103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 --2- 172.21.15.16:0/1508618239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c4103f90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009900 tx=0x7f80b402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 -- 172.21.15.16:0/1508618239 shutdown_connections 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 --2- 172.21.15.16:0/1508618239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4109e10 0x7f80c4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 --2- 172.21.15.16:0/1508618239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c41094c0 0x7f80c41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 --2- 172.21.15.16:0/1508618239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c4103f90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 -- 172.21.15.16:0/1508618239 >> 172.21.15.16:0/1508618239 conn(0x7f80c40ff610 msgr2=0x7f80c4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 -- 172.21.15.16:0/1508618239 shutdown_connections 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.140+0000 7f80cab85640 1 -- 172.21.15.16:0/1508618239 wait complete. 2024-09-21T02:06:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.141+0000 7f80cab85640 1 Processor -- start 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.141+0000 7f80cab85640 1 -- start start 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 0x7f80c419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c4109e10 0x7f80c419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4113980 con 0x7f80c4103b80 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4113800 con 0x7f80c41094c0 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4113b00 con 0x7f80c4109e10 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43022/0 (socket says 172.21.15.16:43022) 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 -- 172.21.15.16:0/2132119305 learned_addr learned my addr 172.21.15.16:0/2132119305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c9122640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c4109e10 0x7f80c419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c3fff640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 0x7f80c419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 -- 172.21.15.16:0/2132119305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c4109e10 msgr2=0x7f80c419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c4109e10 0x7f80c419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 -- 172.21.15.16:0/2132119305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 msgr2=0x7f80c419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 0x7f80c419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 -- 172.21.15.16:0/2132119305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c41a0280 con 0x7f80c4103b80 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c3fff640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 0x7f80c419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c8921640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009a20 tx=0x7f80b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:07.294 INFO:teuthology.orchestra.run.smithi016.stdout:231928234030 2024-09-21T02:06:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b403f030 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f80b40035f0 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c41a0510 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.142+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4035070 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.143+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c41a0a40 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.144+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c4103f90 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.145+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f80b4003aa0 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.146+0000 7f80c1ffb640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 0x7f80ac0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.146+0000 7f80c3fff640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 0x7f80ac0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.146+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f80b41088f0 con 0x7f80c4103b80 2024-09-21T02:06:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.147+0000 7f80c3fff640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 0x7f80ac0b8bb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f80c4109050 tx=0x7f80b8005d30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.150+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f80c41a0a40 con 0x7f80c4103b80 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.339+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f80c419b500 con 0x7f80c4103b80 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.340+0000 7f80c1ffb640 1 -- 172.21.15.16:0/2132119305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f80b40ceb30 con 0x7f80c4103b80 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 msgr2=0x7f80ac0b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 0x7f80ac0b8bb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f80c4109050 tx=0x7f80b8005d30 comp rx=0 tx=0).stop 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 msgr2=0x7f80c419a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009a20 tx=0x7f80b4004270 comp rx=0 tx=0).stop 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 shutdown_connections 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f80ac0b66a0 0x7f80ac0b8bb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80c4109e10 0x7f80c419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c41094c0 0x7f80c419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.342+0000 7f80cab85640 1 --2- 172.21.15.16:0/2132119305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c4103b80 0x7f80c419a3d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.343+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 >> 172.21.15.16:0/2132119305 conn(0x7f80c40ff610 msgr2=0x7f80c4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.343+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 shutdown_connections 2024-09-21T02:06:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:04.343+0000 7f80cab85640 1 -- 172.21.15.16:0/2132119305 wait complete. 2024-09-21T02:06:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:07.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:07 smithi016 ceph-mon[31245]: pgmap v321: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T02:06:07.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234028 got 231928234030 for osd.8 2024-09-21T02:06:07.958 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:08.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:07 smithi123 ceph-mon[33453]: pgmap v321: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T02:06:08.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:07 smithi161 ceph-mon[34061]: pgmap v321: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-21T02:06:08.512 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705749 got 38654705752 for osd.0 2024-09-21T02:06:08.512 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 -- 172.21.15.16:0/3240327317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 msgr2=0x7f87e410d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 --2- 172.21.15.16:0/3240327317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e410d890 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009900 tx=0x7f87d802a410 comp rx=0 tx=0).stop 2024-09-21T02:06:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 -- 172.21.15.16:0/3240327317 shutdown_connections 2024-09-21T02:06:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 --2- 172.21.15.16:0/3240327317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e4110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 --2- 172.21.15.16:0/3240327317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e410d890 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 --2- 172.21.15.16:0/3240327317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 0x7f87e4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 -- 172.21.15.16:0/3240327317 >> 172.21.15.16:0/3240327317 conn(0x7f87e40797e0 msgr2=0x7f87e4079bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 -- 172.21.15.16:0/3240327317 shutdown_connections 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 -- 172.21.15.16:0/3240327317 wait complete. 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.974+0000 7f87e9278640 1 Processor -- start 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 -- start start 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 0x7f87e4102730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e4113960 con 0x7f87e4105340 2024-09-21T02:06:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87e41137e0 con 0x7f87e41049f0 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e9278640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f87e4113ae0 con 0x7f87e410de50 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49918/0 (socket says 172.21.15.16:49918) 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 -- 172.21.15.16:0/3971560696 learned_addr learned my addr 172.21.15.16:0/3971560696 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e2ffd640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 0x7f87e4102730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e37fe640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 -- 172.21.15.16:0/3971560696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 msgr2=0x7f87e41031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e41031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 -- 172.21.15.16:0/3971560696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 msgr2=0x7f87e4102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 0x7f87e4102730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e27fc640 1 -- 172.21.15.16:0/3971560696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e41a85c0 con 0x7f87e4105340 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.975+0000 7f87e37fe640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87e27fc640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87d8005940 tx=0x7f87d80040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d803f030 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f87d8003a80 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8035070 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e41a8850 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.976+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e41a8ce0 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.977+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e4104e00 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.979+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f87d8042030 con 0x7f87e4105340 2024-09-21T02:06:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.979+0000 7f87cbfff640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 0x7f87cc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.979+0000 7f87e2ffd640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 0x7f87cc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.979+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f87d8108230 con 0x7f87e4105340 2024-09-21T02:06:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.980+0000 7f87e2ffd640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 0x7f87cc0b8b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f87e4104180 tx=0x7f87d4009670 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:08.983+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f87d80ce4d0 con 0x7f87e4105340 2024-09-21T02:06:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.179+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f87e402e640 con 0x7f87e4105340 2024-09-21T02:06:09.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.179+0000 7f87cbfff640 1 -- 172.21.15.16:0/3971560696 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f87d80d28b0 con 0x7f87e4105340 2024-09-21T02:06:09.180 INFO:teuthology.orchestra.run.smithi016.stdout:210453397556 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 msgr2=0x7f87cc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 0x7f87cc0b8b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f87e4104180 tx=0x7f87d4009670 comp rx=0 tx=0).stop 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 msgr2=0x7f87e4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87d8005940 tx=0x7f87d80040e0 comp rx=0 tx=0).stop 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 shutdown_connections 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f87cc0b6670 0x7f87cc0b8b80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87e410de50 0x7f87e41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e4105340 0x7f87e4102c70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 --2- 172.21.15.16:0/3971560696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e41049f0 0x7f87e4102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 >> 172.21.15.16:0/3971560696 conn(0x7f87e40797e0 msgr2=0x7f87e4100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 shutdown_connections 2024-09-21T02:06:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:09.182+0000 7f87e9278640 1 -- 172.21.15.16:0/3971560696 wait complete. 2024-09-21T02:06:10.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:09 smithi123 ceph-mon[33453]: pgmap v322: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-21T02:06:10.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:09 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3971560696' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:06:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:09 smithi016 ceph-mon[31245]: pgmap v322: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-21T02:06:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:09 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3971560696' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:06:10.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:09 smithi161 ceph-mon[34061]: pgmap v322: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-21T02:06:10.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:09 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3971560696' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T02:06:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.203+0000 7f0140171640 1 -- 172.21.15.16:0/674401517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01381083d0 msgr2=0x7f01381087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.203+0000 7f0140171640 1 --2- 172.21.15.16:0/674401517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01381083d0 0x7f01381087e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f012c009900 tx=0x7f012c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 -- 172.21.15.16:0/674401517 shutdown_connections 2024-09-21T02:06:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 --2- 172.21.15.16:0/674401517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0138108d20 0x7f013810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 --2- 172.21.15.16:0/674401517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01381083d0 0x7f01381087e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 --2- 172.21.15.16:0/674401517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 0x7f0138102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 -- 172.21.15.16:0/674401517 >> 172.21.15.16:0/674401517 conn(0x7f0138079760 msgr2=0x7f0138079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 -- 172.21.15.16:0/674401517 shutdown_connections 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 -- 172.21.15.16:0/674401517 wait complete. 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 Processor -- start 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.204+0000 7f0140171640 1 -- start start 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 0x7f013819c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01381083d0 0x7f013819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0138112840 con 0x7f0138108d20 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01381126c0 con 0x7f01381083d0 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f0140171640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01381129c0 con 0x7f0138102ae0 2024-09-21T02:06:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49930/0 (socket says 172.21.15.16:49930) 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 -- 172.21.15.16:0/65269622 learned_addr learned my addr 172.21.15.16:0/65269622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013df0d640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 0x7f013819c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013d70c640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01381083d0 0x7f013819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 -- 172.21.15.16:0/65269622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 msgr2=0x7f013819c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 0x7f013819c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 -- 172.21.15.16:0/65269622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01381083d0 msgr2=0x7f013819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01381083d0 0x7f013819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 -- 172.21.15.16:0/65269622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01381a2530 con 0x7f0138108d20 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.205+0000 7f013e70e640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f013000eaa0 tx=0x7f013000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:10.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.206+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000cdb0 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.206+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01381a2820 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.206+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0130005020 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.206+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01300116d0 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.206+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01381a8f80 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.207+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0138103d10 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.208+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f01300051d0 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.209+0000 7f012affd640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 0x7f01240b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.209+0000 7f013df0d640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 0x7f01240b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.209+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f01300e7ff0 con 0x7f0138108d20 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.210+0000 7f013df0d640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 0x7f01240b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0134005fd0 tx=0x7f013400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.212+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0130016030 con 0x7f0138108d20 2024-09-21T02:06:10.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.402+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0138102ef0 con 0x7f0138108d20 2024-09-21T02:06:10.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.403+0000 7f012affd640 1 -- 172.21.15.16:0/65269622 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0138102ef0 con 0x7f0138108d20 2024-09-21T02:06:10.404 INFO:teuthology.orchestra.run.smithi016.stdout:188978561082 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 msgr2=0x7f01240b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 0x7f01240b8b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0134005fd0 tx=0x7f013400b390 comp rx=0 tx=0).stop 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 msgr2=0x7f01381a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f013000eaa0 tx=0x7f013000ef60 comp rx=0 tx=0).stop 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 shutdown_connections 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f01240b65f0 0x7f01240b8b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0138108d20 0x7f01381a1e00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01381083d0 0x7f013819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 --2- 172.21.15.16:0/65269622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0138102ae0 0x7f013819c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 >> 172.21.15.16:0/65269622 conn(0x7f0138079760 msgr2=0x7f01381043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:10.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 shutdown_connections 2024-09-21T02:06:10.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.405+0000 7f0140171640 1 -- 172.21.15.16:0/65269622 wait complete. 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 -- 172.21.15.16:0/3194127408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0102ae0 msgr2=0x7f3ac0102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/3194127408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0102ae0 0x7f3ac0102ef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009900 tx=0x7f3abc02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 -- 172.21.15.16:0/3194127408 shutdown_connections 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/3194127408 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac0108d20 0x7f3ac010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/3194127408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac01083d0 0x7f3ac01087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/3194127408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0102ae0 0x7f3ac0102ef0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 -- 172.21.15.16:0/3194127408 >> 172.21.15.16:0/3194127408 conn(0x7f3ac0079760 msgr2=0x7f3ac0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 -- 172.21.15.16:0/3194127408 shutdown_connections 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.567+0000 7f3ac85fc640 1 -- 172.21.15.16:0/3194127408 wait complete. 2024-09-21T02:06:10.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 Processor -- start 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 -- start start 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac0102ae0 0x7f3ac019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac01083d0 0x7f3ac019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0112840 con 0x7f3ac0108d20 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac01126c0 con 0x7f3ac0102ae0 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.568+0000 7f3ac85fc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac01129c0 con 0x7f3ac01083d0 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49938/0 (socket says 172.21.15.16:49938) 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 -- 172.21.15.16:0/4093209661 learned_addr learned my addr 172.21.15.16:0/4093209661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6398640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac0102ae0 0x7f3ac019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac5b97640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac01083d0 0x7f3ac019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 -- 172.21.15.16:0/4093209661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac01083d0 msgr2=0x7f3ac019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac01083d0 0x7f3ac019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 -- 172.21.15.16:0/4093209661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac0102ae0 msgr2=0x7f3ac019b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac0102ae0 0x7f3ac019b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 -- 172.21.15.16:0/4093209661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac01a6440 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac6b99640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800eaa0 tx=0x7f3ab800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab800cdb0 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ab8005020 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac01a6730 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab80116d0 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.569+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac01a6c70 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.570+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac0103d10 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.572+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f3ab80051d0 con 0x7f3ac0108d20 2024-09-21T02:06:10.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.573+0000 7f3ab77fe640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 0x7f3aac0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.573+0000 7f3ac6398640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 0x7f3aac0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.573+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f3ab80e7e40 con 0x7f3ac0108d20 2024-09-21T02:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.573+0000 7f3ac6398640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 0x7f3aac0b8bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ac006c6a0 tx=0x7f3abc0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:10.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.576+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3ab8016030 con 0x7f3ac0108d20 2024-09-21T02:06:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.769+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f3ac0102ef0 con 0x7f3ac0108d20 2024-09-21T02:06:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.769+0000 7f3ab77fe640 1 -- 172.21.15.16:0/4093209661 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f3ac0102ef0 con 0x7f3ac0108d20 2024-09-21T02:06:10.770 INFO:teuthology.orchestra.run.smithi016.stdout:279172874277 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 msgr2=0x7f3aac0b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 0x7f3aac0b8bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ac006c6a0 tx=0x7f3abc0058d0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 msgr2=0x7f3ac019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800eaa0 tx=0x7f3ab800ef60 comp rx=0 tx=0).stop 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 shutdown_connections 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3aac0b66a0 0x7f3aac0b8bb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ac0108d20 0x7f3ac019bc30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac01083d0 0x7f3ac019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 --2- 172.21.15.16:0/4093209661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac0102ae0 0x7f3ac019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:10.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 >> 172.21.15.16:0/4093209661 conn(0x7f3ac0079760 msgr2=0x7f3ac0101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:10.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 shutdown_connections 2024-09-21T02:06:10.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:10.771+0000 7f3ac85fc640 1 -- 172.21.15.16:0/4093209661 wait complete. 2024-09-21T02:06:11.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:10 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/65269622' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:06:11.058 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:10 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/65269622' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:06:11.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:10 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/65269622' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T02:06:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.356+0000 7f6674edd640 1 -- 172.21.15.16:0/1251016570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66701094a0 msgr2=0x7f66701098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.356+0000 7f6674edd640 1 --2- 172.21.15.16:0/1251016570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66701094a0 0x7f66701098b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6664009900 tx=0x7f666402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 -- 172.21.15.16:0/1251016570 shutdown_connections 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 --2- 172.21.15.16:0/1251016570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670109df0 0x7f6670110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 --2- 172.21.15.16:0/1251016570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66701094a0 0x7f66701098b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 --2- 172.21.15.16:0/1251016570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670103b60 0x7f6670103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 -- 172.21.15.16:0/1251016570 >> 172.21.15.16:0/1251016570 conn(0x7f667007b850 msgr2=0x7f667007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 -- 172.21.15.16:0/1251016570 shutdown_connections 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.357+0000 7f6674edd640 1 -- 172.21.15.16:0/1251016570 wait complete. 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 Processor -- start 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 -- start start 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66701094a0 0x7f667019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670109df0 0x7f66701a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6670113960 con 0x7f6670103b60 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66701137e0 con 0x7f66701094a0 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.358+0000 7f6674edd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6670113ae0 con 0x7f6670109df0 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49950/0 (socket says 172.21.15.16:49950) 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 -- 172.21.15.16:0/337021970 learned_addr learned my addr 172.21.15.16:0/337021970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666dd9b640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66701094a0 0x7f667019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666ed9d640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670109df0 0x7f66701a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 -- 172.21.15.16:0/337021970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670109df0 msgr2=0x7f66701a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670109df0 0x7f66701a1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 -- 172.21.15.16:0/337021970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66701094a0 msgr2=0x7f667019cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66701094a0 0x7f667019cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 -- 172.21.15.16:0/337021970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66701a25c0 con 0x7f6670103b60 2024-09-21T02:06:11.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f666e59c640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f666000d950 tx=0x7f666000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660017070 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6660009c30 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66701a28b0 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.359+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666000f620 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.360+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66701a9010 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.360+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6670103f70 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.362+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f6660005030 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.363+0000 7f665f7fe640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 0x7f66580b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.363+0000 7f666dd9b640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 0x7f66580b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.363+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f66600e7460 con 0x7f6670103b60 2024-09-21T02:06:11.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.364+0000 7f666dd9b640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 0x7f66580b8b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6664005940 tx=0x7f66640058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.367+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f66600ad820 con 0x7f6670103b60 2024-09-21T02:06:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.560+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f66701a92f0 con 0x7f6670103b60 2024-09-21T02:06:11.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.560+0000 7f665f7fe640 1 -- 172.21.15.16:0/337021970 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f6660009de0 con 0x7f6670103b60 2024-09-21T02:06:11.561 INFO:teuthology.orchestra.run.smithi016.stdout:300647710753 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.562+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 msgr2=0x7f66580b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.562+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 0x7f66580b8b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6664005940 tx=0x7f66640058d0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.562+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 msgr2=0x7f667019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.562+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f666000d950 tx=0x7f666000de10 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.562+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 shutdown_connections 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66580b6670 0x7f66580b8b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6670109df0 0x7f66701a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66701094a0 0x7f667019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 --2- 172.21.15.16:0/337021970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670103b60 0x7f667019c650 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 >> 172.21.15.16:0/337021970 conn(0x7f667007b850 msgr2=0x7f66701073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 shutdown_connections 2024-09-21T02:06:11.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:11.563+0000 7f6674edd640 1 -- 172.21.15.16:0/337021970 wait complete. 2024-09-21T02:06:11.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397554 got 210453397556 for osd.7 2024-09-21T02:06:11.823 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:11 smithi123 ceph-mon[33453]: pgmap v323: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-09-21T02:06:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4093209661' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:06:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/337021970' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:06:12.154 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:11 smithi016 ceph-mon[31245]: pgmap v323: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-09-21T02:06:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4093209661' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:06:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/337021970' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:06:12.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:11 smithi161 ceph-mon[34061]: pgmap v323: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-09-21T02:06:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4093209661' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-21T02:06:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/337021970' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 -- 172.21.15.16:0/2192188473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc1094a0 msgr2=0x7f98dc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 --2- 172.21.15.16:0/2192188473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc1094a0 0x7f98dc1098b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009900 tx=0x7f98c402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 -- 172.21.15.16:0/2192188473 shutdown_connections 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 --2- 172.21.15.16:0/2192188473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 --2- 172.21.15.16:0/2192188473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc1094a0 0x7f98dc1098b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 --2- 172.21.15.16:0/2192188473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc103b60 0x7f98dc103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 -- 172.21.15.16:0/2192188473 >> 172.21.15.16:0/2192188473 conn(0x7f98dc07b850 msgr2=0x7f98dc07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 -- 172.21.15.16:0/2192188473 shutdown_connections 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.460+0000 7f98e205a640 1 -- 172.21.15.16:0/2192188473 wait complete. 2024-09-21T02:06:12.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.461+0000 7f98e205a640 1 Processor -- start 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 -- start start 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc1094a0 0x7f98dc19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc113960 con 0x7f98dc103b60 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc1137e0 con 0x7f98dc1094a0 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98e205a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc113ae0 con 0x7f98dc109df0 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49966/0 (socket says 172.21.15.16:49966) 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 -- 172.21.15.16:0/2451877333 learned_addr learned my addr 172.21.15.16:0/2451877333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98d3fff640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc1094a0 0x7f98dc19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98dbfff640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 -- 172.21.15.16:0/2451877333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 msgr2=0x7f98dc19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:12.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 -- 172.21.15.16:0/2451877333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc1094a0 msgr2=0x7f98dc19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc1094a0 0x7f98dc19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98db7fe640 1 -- 172.21.15.16:0/2451877333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98dc1a03c0 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.462+0000 7f98dbfff640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98db7fe640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00d950 tx=0x7f98cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc017070 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98cc009c30 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98dc1a06b0 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00f620 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.463+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98dc1a6e10 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.464+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98dc103f70 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.465+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f98cc005030 con 0x7f98dc103b60 2024-09-21T02:06:12.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.466+0000 7f98d97fa640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 0x7f98c80b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.467+0000 7f98d3fff640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 0x7f98c80b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.467+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f98cc0e75d0 con 0x7f98dc103b60 2024-09-21T02:06:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.467+0000 7f98d3fff640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 0x7f98c80b8b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f98c4005940 tx=0x7f98c40058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:12.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.470+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f98cc0ad810 con 0x7f98dc103b60 2024-09-21T02:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.661+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f98dc19b580 con 0x7f98dc103b60 2024-09-21T02:06:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.661+0000 7f98d97fa640 1 -- 172.21.15.16:0/2451877333 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f98dc19b580 con 0x7f98dc103b60 2024-09-21T02:06:12.662 INFO:teuthology.orchestra.run.smithi016.stdout:257698037803 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 msgr2=0x7f98c80b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 0x7f98c80b8b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f98c4005940 tx=0x7f98c40058d0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 msgr2=0x7f98dc19a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00d950 tx=0x7f98cc00de10 comp rx=0 tx=0).stop 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 shutdown_connections 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f98c80b65f0 0x7f98c80b8b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f98dc109df0 0x7f98dc19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98dc1094a0 0x7f98dc19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 --2- 172.21.15.16:0/2451877333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98dc103b60 0x7f98dc19a450 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 >> 172.21.15.16:0/2451877333 conn(0x7f98dc07b850 msgr2=0x7f98dc105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 shutdown_connections 2024-09-21T02:06:12.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:12.663+0000 7f98e205a640 1 -- 172.21.15.16:0/2451877333 wait complete. 2024-09-21T02:06:13.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:12 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2451877333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:06:13.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:12 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2451877333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:06:13.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:12 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2451877333' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-21T02:06:13.458 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037800 got 257698037803 for osd.9 2024-09-21T02:06:13.458 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:13.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710751 got 300647710753 for osd.11 2024-09-21T02:06:13.907 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:13 smithi016 ceph-mon[31245]: pgmap v324: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:14.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:13 smithi161 ceph-mon[34061]: pgmap v324: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:14.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874274 got 279172874277 for osd.10 2024-09-21T02:06:14.258 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:13 smithi123 ceph-mon[33453]: pgmap v324: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:14.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561079 got 188978561082 for osd.6 2024-09-21T02:06:14.724 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 -- 172.21.15.16:0/220846138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c1094a0 msgr2=0x7fd16c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 --2- 172.21.15.16:0/220846138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c1094a0 0x7fd16c1098b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 -- 172.21.15.16:0/220846138 shutdown_connections 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 --2- 172.21.15.16:0/220846138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c109df0 0x7fd16c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 --2- 172.21.15.16:0/220846138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c1094a0 0x7fd16c1098b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 --2- 172.21.15.16:0/220846138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 0x7fd16c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 -- 172.21.15.16:0/220846138 >> 172.21.15.16:0/220846138 conn(0x7fd16c07b850 msgr2=0x7fd16c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 -- 172.21.15.16:0/220846138 shutdown_connections 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.283+0000 7fd1738f0640 1 -- 172.21.15.16:0/220846138 wait complete. 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 Processor -- start 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 -- start start 2024-09-21T02:06:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 0x7fd16c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 0x7fd16c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c113960 con 0x7fd16c109df0 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1137e0 con 0x7fd16c1094a0 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd1738f0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c113ae0 con 0x7fd16c103b60 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd170e8b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd170e8b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:47128/0 (socket says 172.21.15.16:47128) 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.284+0000 7fd170e8b640 1 -- 172.21.15.16:0/2461095319 learned_addr learned my addr 172.21.15.16:0/2461095319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd17168c640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 0x7fd16c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd171e8d640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 0x7fd16c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 -- 172.21.15.16:0/2461095319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 msgr2=0x7fd16c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 0x7fd16c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 -- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 msgr2=0x7fd16c076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 0x7fd16c076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 -- 172.21.15.16:0/2461095319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1a8670 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd171e8d640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 0x7fd16c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd170e8b640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005940 tx=0x7fd15c0040e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c03f030 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1a8900 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd15c003a80 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.285+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c035070 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.286+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1a8de0 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.287+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c103f70 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.289+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd15c042030 con 0x7fd16c1094a0 2024-09-21T02:06:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.289+0000 7fd1627fc640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 0x7fd1580b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.289+0000 7fd17168c640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 0x7fd1580b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.289+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fd15c108200 con 0x7fd16c1094a0 2024-09-21T02:06:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.290+0000 7fd17168c640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 0x7fd1580b8b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd16c077160 tx=0x7fd16800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:15.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.293+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd15c0d2880 con 0x7fd16c1094a0 2024-09-21T02:06:15.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.484+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd16c02e640 con 0x7fd16c1094a0 2024-09-21T02:06:15.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.484+0000 7fd1627fc640 1 -- 172.21.15.16:0/2461095319 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd15c0d77c0 con 0x7fd16c1094a0 2024-09-21T02:06:15.485 INFO:teuthology.orchestra.run.smithi016.stdout:141733920837 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 msgr2=0x7fd1580b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 0x7fd1580b8b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd16c077160 tx=0x7fd16800b410 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 msgr2=0x7fd16c075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005940 tx=0x7fd15c0040e0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 shutdown_connections 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd1580b65f0 0x7fd1580b8b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd16c109df0 0x7fd16c076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd16c1094a0 0x7fd16c075c50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 --2- 172.21.15.16:0/2461095319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd16c103b60 0x7fd16c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 >> 172.21.15.16:0/2461095319 conn(0x7fd16c07b850 msgr2=0x7fd16c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 shutdown_connections 2024-09-21T02:06:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:15.487+0000 7fd1738f0640 1 -- 172.21.15.16:0/2461095319 wait complete. 2024-09-21T02:06:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:15 smithi016 ceph-mon[31245]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:15 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2461095319' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:06:16.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:15 smithi161 ceph-mon[34061]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:16.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:15 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2461095319' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:06:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:15 smithi123 ceph-mon[33453]: pgmap v325: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:15 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2461095319' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T02:06:16.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920833 got 141733920837 for osd.4 2024-09-21T02:06:16.338 DEBUG:teuthology.parallel:result is None 2024-09-21T02:06:16.339 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:06:16.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:06:16.605 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 -- 172.21.15.16:0/2355584747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01083d0 msgr2=0x7f30c01087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:17.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 --2- 172.21.15.16:0/2355584747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01083d0 0x7f30c01087e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009900 tx=0x7f30b402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 -- 172.21.15.16:0/2355584747 shutdown_connections 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 --2- 172.21.15.16:0/2355584747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c0108d20 0x7f30c010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 --2- 172.21.15.16:0/2355584747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01083d0 0x7f30c01087e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 --2- 172.21.15.16:0/2355584747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0102ae0 0x7f30c0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 -- 172.21.15.16:0/2355584747 >> 172.21.15.16:0/2355584747 conn(0x7f30c0079760 msgr2=0x7f30c0079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 -- 172.21.15.16:0/2355584747 shutdown_connections 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.098+0000 7f30c5bea640 1 -- 172.21.15.16:0/2355584747 wait complete. 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.099+0000 7f30c5bea640 1 Processor -- start 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.099+0000 7f30c5bea640 1 -- start start 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c01083d0 0x7f30c0199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 0x7f30c0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30c0112840 con 0x7f30c0102ae0 2024-09-21T02:06:17.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30c01126c0 con 0x7f30c0108d20 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30c5bea640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f30c01129c0 con 0x7f30c01083d0 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49990/0 (socket says 172.21.15.16:49990) 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 -- 172.21.15.16:0/3634013805 learned_addr learned my addr 172.21.15.16:0/3634013805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bffff640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 0x7f30c0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30beffd640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c01083d0 0x7f30c0199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 -- 172.21.15.16:0/3634013805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c01083d0 msgr2=0x7f30c0199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c01083d0 0x7f30c0199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 -- 172.21.15.16:0/3634013805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 msgr2=0x7f30c0199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 0x7f30c0199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 -- 172.21.15.16:0/3634013805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c01a8640 con 0x7f30c0102ae0 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bffff640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 0x7f30c0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:17.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bf7fe640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c920 tx=0x7f30b000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.100+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0007c70 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.101+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30b0007e10 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.101+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c01a88d0 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.101+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0018710 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.101+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c01a8e30 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.102+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c0103d10 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.103+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f30b000f450 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.104+0000 7f30bcff9640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 0x7f30a80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.104+0000 7f30beffd640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 0x7f30a80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.104+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f30b00e7e30 con 0x7f30c0102ae0 2024-09-21T02:06:17.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.105+0000 7f30beffd640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 0x7f30a80b8b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f30b4006010 tx=0x7f30b4031c70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.108+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f30b0014030 con 0x7f30c0102ae0 2024-09-21T02:06:17.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.295+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f30c0063800 con 0x7f30a80b6670 2024-09-21T02:06:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.301+0000 7f30bcff9640 1 -- 172.21.15.16:0/3634013805 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340648 (secure 0 0 0) 0x7f30c0063800 con 0x7f30a80b6670 2024-09-21T02:06:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:06:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 msgr2=0x7f30a80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 0x7f30a80b8b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f30b4006010 tx=0x7f30b4031c70 comp rx=0 tx=0).stop 2024-09-21T02:06:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 msgr2=0x7f30c0198fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c920 tx=0x7f30b000cde0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 shutdown_connections 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30a80b6670 0x7f30a80b8b80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30c0108d20 0x7f30c0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30c01083d0 0x7f30c0199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 --2- 172.21.15.16:0/3634013805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0102ae0 0x7f30c0198fc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 >> 172.21.15.16:0/3634013805 conn(0x7f30c0079760 msgr2=0x7f30c01044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 shutdown_connections 2024-09-21T02:06:17.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:17.305+0000 7f30c5bea640 1 -- 172.21.15.16:0/3634013805 wait complete. 2024-09-21T02:06:18.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:17 smithi016 ceph-mon[31245]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:18.161 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-21T02:06:16.043999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664800,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":657,"num_read_kb":438,"num_write":376,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394292,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547532,"statfs":{"total":1151940427776,"available":1151536672768,"internally_reserved":0,"allocated":11976704,"data_stored":7968002,"data_compressed":465210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003936"},"pg_stats":[{"pgid":"2.1f","version":"84'13","reported_seq":105,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:10.819160+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:06:10.819160+0000","last_peered":"2024-09-21T02:06:10.819160+0000","last_clean":"2024-09-21T02:06:10.819160+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:06:10.819160+0000","last_undegraded":"2024-09-21T02:06:10.819160+0000","last_fullsized":"2024-09-21T02:06:10.819160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693911+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:05:54.693911+0000","last_peered":"2024-09-21T02:05:54.693911+0000","last_clean":"2024-09-21T02:05:54.693911+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:05:54.693911+0000","last_undegraded":"2024-09-21T02:05:54.693911+0000","last_fullsized":"2024-09-21T02:05:54.693911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309473+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:05:54.309473+0000","last_peered":"2024-09-21T02:05:54.309473+0000","last_clean":"2024-09-21T02:05:54.309473+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:05:54.309473+0000","last_undegraded":"2024-09-21T02:05:54.309473+0000","last_fullsized":"2024-09-21T02:05:54.309473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694295+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:05:54.694295+0000","last_peered":"2024-09-21T02:05:54.694295+0000","last_clean":"2024-09-21T02:05:54.694295+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:05:54.694295+0000","last_undegraded":"2024-09-21T02:05:54.694295+0000","last_fullsized":"2024-09-21T02:05:54.694295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838888+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:05:54.838888+0000","last_peered":"2024-09-21T02:05:54.838888+0000","last_clean":"2024-09-21T02:05:54.838888+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:05:54.838888+0000","last_undegraded":"2024-09-21T02:05:54.838888+0000","last_fullsized":"2024-09-21T02:05:54.838888+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313792+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:05:54.313792+0000","last_peered":"2024-09-21T02:05:54.313792+0000","last_clean":"2024-09-21T02:05:54.313792+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:05:54.313792+0000","last_undegraded":"2024-09-21T02:05:54.313792+0000","last_fullsized":"2024-09-21T02:05:54.313792+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314180+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:54.314180+0000","last_peered":"2024-09-21T02:05:54.314180+0000","last_clean":"2024-09-21T02:05:54.314180+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:54.314180+0000","last_undegraded":"2024-09-21T02:05:54.314180+0000","last_fullsized":"2024-09-21T02:05:54.314180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839131+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:05:54.839131+0000","last_peered":"2024-09-21T02:05:54.839131+0000","last_clean":"2024-09-21T02:05:54.839131+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:05:54.839131+0000","last_undegraded":"2024-09-21T02:05:54.839131+0000","last_fullsized":"2024-09-21T02:05:54.839131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841186+0000","last_change":"2024-09-21T02:05:54.841083+0000","last_active":"2024-09-21T02:05:54.841186+0000","last_peered":"2024-09-21T02:05:54.841186+0000","last_clean":"2024-09-21T02:05:54.841186+0000","last_became_active":"2024-09-21T02:05:54.840645+0000","last_became_peered":"2024-09-21T02:05:54.840645+0000","last_unstale":"2024-09-21T02:05:54.841186+0000","last_undegraded":"2024-09-21T02:05:54.841186+0000","last_fullsized":"2024-09-21T02:05:54.841186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:33:35.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840245+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:54.840245+0000","last_peered":"2024-09-21T02:05:54.840245+0000","last_clean":"2024-09-21T02:05:54.840245+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:54.840245+0000","last_undegraded":"2024-09-21T02:05:54.840245+0000","last_fullsized":"2024-09-21T02:05:54.840245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795801+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:05:54.795801+0000","last_peered":"2024-09-21T02:05:54.795801+0000","last_clean":"2024-09-21T02:05:54.795801+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:05:54.795801+0000","last_undegraded":"2024-09-21T02:05:54.795801+0000","last_fullsized":"2024-09-21T02:05:54.795801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313927+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:05:54.313927+0000","last_peered":"2024-09-21T02:05:54.313927+0000","last_clean":"2024-09-21T02:05:54.313927+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:05:54.313927+0000","last_undegraded":"2024-09-21T02:05:54.313927+0000","last_fullsized":"2024-09-21T02:05:54.313927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799300+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:05:54.799300+0000","last_peered":"2024-09-21T02:05:54.799300+0000","last_clean":"2024-09-21T02:05:54.799300+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:05:54.799300+0000","last_undegraded":"2024-09-21T02:05:54.799300+0000","last_fullsized":"2024-09-21T02:05:54.799300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839857+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:54.839857+0000","last_peered":"2024-09-21T02:05:54.839857+0000","last_clean":"2024-09-21T02:05:54.839857+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:54.839857+0000","last_undegraded":"2024-09-21T02:05:54.839857+0000","last_fullsized":"2024-09-21T02:05:54.839857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314162+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:05:54.314162+0000","last_peered":"2024-09-21T02:05:54.314162+0000","last_clean":"2024-09-21T02:05:54.314162+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:05:54.314162+0000","last_undegraded":"2024-09-21T02:05:54.314162+0000","last_fullsized":"2024-09-21T02:05:54.314162+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693857+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:05:54.693857+0000","last_peered":"2024-09-21T02:05:54.693857+0000","last_clean":"2024-09-21T02:05:54.693857+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:05:54.693857+0000","last_undegraded":"2024-09-21T02:05:54.693857+0000","last_fullsized":"2024-09-21T02:05:54.693857+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309822+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:05:54.309822+0000","last_peered":"2024-09-21T02:05:54.309822+0000","last_clean":"2024-09-21T02:05:54.309822+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:05:54.309822+0000","last_undegraded":"2024-09-21T02:05:54.309822+0000","last_fullsized":"2024-09-21T02:05:54.309822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314166+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:05:54.314166+0000","last_peered":"2024-09-21T02:05:54.314166+0000","last_clean":"2024-09-21T02:05:54.314166+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:05:54.314166+0000","last_undegraded":"2024-09-21T02:05:54.314166+0000","last_fullsized":"2024-09-21T02:05:54.314166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313914+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:05:54.313914+0000","last_peered":"2024-09-21T02:05:54.313914+0000","last_clean":"2024-09-21T02:05:54.313914+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:05:54.313914+0000","last_undegraded":"2024-09-21T02:05:54.313914+0000","last_fullsized":"2024-09-21T02:05:54.313914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693836+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:54.693836+0000","last_peered":"2024-09-21T02:05:54.693836+0000","last_clean":"2024-09-21T02:05:54.693836+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:54.693836+0000","last_undegraded":"2024-09-21T02:05:54.693836+0000","last_fullsized":"2024-09-21T02:05:54.693836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.754062+0000","last_change":"2024-09-21T02:05:54.753929+0000","last_active":"2024-09-21T02:05:54.754062+0000","last_peered":"2024-09-21T02:05:54.754062+0000","last_clean":"2024-09-21T02:05:54.754062+0000","last_became_active":"2024-09-21T02:05:54.753431+0000","last_became_peered":"2024-09-21T02:05:54.753431+0000","last_unstale":"2024-09-21T02:05:54.754062+0000","last_undegraded":"2024-09-21T02:05:54.754062+0000","last_fullsized":"2024-09-21T02:05:54.754062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:20:41.311972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839451+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:05:54.839451+0000","last_peered":"2024-09-21T02:05:54.839451+0000","last_clean":"2024-09-21T02:05:54.839451+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:05:54.839451+0000","last_undegraded":"2024-09-21T02:05:54.839451+0000","last_fullsized":"2024-09-21T02:05:54.839451+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839025+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:54.839025+0000","last_peered":"2024-09-21T02:05:54.839025+0000","last_clean":"2024-09-21T02:05:54.839025+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:54.839025+0000","last_undegraded":"2024-09-21T02:05:54.839025+0000","last_fullsized":"2024-09-21T02:05:54.839025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314212+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:05:54.314212+0000","last_peered":"2024-09-21T02:05:54.314212+0000","last_clean":"2024-09-21T02:05:54.314212+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:05:54.314212+0000","last_undegraded":"2024-09-21T02:05:54.314212+0000","last_fullsized":"2024-09-21T02:05:54.314212+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838738+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:05:54.838738+0000","last_peered":"2024-09-21T02:05:54.838738+0000","last_clean":"2024-09-21T02:05:54.838738+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:05:54.838738+0000","last_undegraded":"2024-09-21T02:05:54.838738+0000","last_fullsized":"2024-09-21T02:05:54.838738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314577+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:54.314577+0000","last_peered":"2024-09-21T02:05:54.314577+0000","last_clean":"2024-09-21T02:05:54.314577+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:54.314577+0000","last_undegraded":"2024-09-21T02:05:54.314577+0000","last_fullsized":"2024-09-21T02:05:54.314577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309620+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:05:54.309620+0000","last_peered":"2024-09-21T02:05:54.309620+0000","last_clean":"2024-09-21T02:05:54.309620+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:05:54.309620+0000","last_undegraded":"2024-09-21T02:05:54.309620+0000","last_fullsized":"2024-09-21T02:05:54.309620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694104+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:05:54.694104+0000","last_peered":"2024-09-21T02:05:54.694104+0000","last_clean":"2024-09-21T02:05:54.694104+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:05:54.694104+0000","last_undegraded":"2024-09-21T02:05:54.694104+0000","last_fullsized":"2024-09-21T02:05:54.694104+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'16","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796179+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:05:54.796179+0000","last_peered":"2024-09-21T02:05:54.796179+0000","last_clean":"2024-09-21T02:05:54.796179+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:05:54.796179+0000","last_undegraded":"2024-09-21T02:05:54.796179+0000","last_fullsized":"2024-09-21T02:05:54.796179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693484+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:05:54.693484+0000","last_peered":"2024-09-21T02:05:54.693484+0000","last_clean":"2024-09-21T02:05:54.693484+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:05:54.693484+0000","last_undegraded":"2024-09-21T02:05:54.693484+0000","last_fullsized":"2024-09-21T02:05:54.693484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308954+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:05:54.308954+0000","last_peered":"2024-09-21T02:05:54.308954+0000","last_clean":"2024-09-21T02:05:54.308954+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:05:54.308954+0000","last_undegraded":"2024-09-21T02:05:54.308954+0000","last_fullsized":"2024-09-21T02:05:54.308954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314094+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:54.314094+0000","last_peered":"2024-09-21T02:05:54.314094+0000","last_clean":"2024-09-21T02:05:54.314094+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:54.314094+0000","last_undegraded":"2024-09-21T02:05:54.314094+0000","last_fullsized":"2024-09-21T02:05:54.314094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313911+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:05:54.313911+0000","last_peered":"2024-09-21T02:05:54.313911+0000","last_clean":"2024-09-21T02:05:54.313911+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:05:54.313911+0000","last_undegraded":"2024-09-21T02:05:54.313911+0000","last_fullsized":"2024-09-21T02:05:54.313911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309554+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:05:54.309554+0000","last_peered":"2024-09-21T02:05:54.309554+0000","last_clean":"2024-09-21T02:05:54.309554+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:05:54.309554+0000","last_undegraded":"2024-09-21T02:05:54.309554+0000","last_fullsized":"2024-09-21T02:05:54.309554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796030+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:05:54.796030+0000","last_peered":"2024-09-21T02:05:54.796030+0000","last_clean":"2024-09-21T02:05:54.796030+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:05:54.796030+0000","last_undegraded":"2024-09-21T02:05:54.796030+0000","last_fullsized":"2024-09-21T02:05:54.796030+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839502+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:05:54.839502+0000","last_peered":"2024-09-21T02:05:54.839502+0000","last_clean":"2024-09-21T02:05:54.839502+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:05:54.839502+0000","last_undegraded":"2024-09-21T02:05:54.839502+0000","last_fullsized":"2024-09-21T02:05:54.839502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309631+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:05:54.309631+0000","last_peered":"2024-09-21T02:05:54.309631+0000","last_clean":"2024-09-21T02:05:54.309631+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:05:54.309631+0000","last_undegraded":"2024-09-21T02:05:54.309631+0000","last_fullsized":"2024-09-21T02:05:54.309631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":103,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:14.825713+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:06:14.825713+0000","last_peered":"2024-09-21T02:06:14.825713+0000","last_clean":"2024-09-21T02:06:14.825713+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:06:14.825713+0000","last_undegraded":"2024-09-21T02:06:14.825713+0000","last_fullsized":"2024-09-21T02:06:14.825713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308803+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:05:54.308803+0000","last_peered":"2024-09-21T02:05:54.308803+0000","last_clean":"2024-09-21T02:05:54.308803+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:05:54.308803+0000","last_undegraded":"2024-09-21T02:05:54.308803+0000","last_fullsized":"2024-09-21T02:05:54.308803+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752017+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:05:54.752017+0000","last_peered":"2024-09-21T02:05:54.752017+0000","last_clean":"2024-09-21T02:05:54.752017+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:05:54.752017+0000","last_undegraded":"2024-09-21T02:05:54.752017+0000","last_fullsized":"2024-09-21T02:05:54.752017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":100,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:13.824149+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:06:13.824149+0000","last_peered":"2024-09-21T02:06:13.824149+0000","last_clean":"2024-09-21T02:06:13.824149+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:06:13.824149+0000","last_undegraded":"2024-09-21T02:06:13.824149+0000","last_fullsized":"2024-09-21T02:06:13.824149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":56,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309175+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:05:54.309175+0000","last_peered":"2024-09-21T02:05:54.309175+0000","last_clean":"2024-09-21T02:05:54.309175+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:05:54.309175+0000","last_undegraded":"2024-09-21T02:05:54.309175+0000","last_fullsized":"2024-09-21T02:05:54.309175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308470+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:05:54.308470+0000","last_peered":"2024-09-21T02:05:54.308470+0000","last_clean":"2024-09-21T02:05:54.308470+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:05:54.308470+0000","last_undegraded":"2024-09-21T02:05:54.308470+0000","last_fullsized":"2024-09-21T02:05:54.308470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799262+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:54.799262+0000","last_peered":"2024-09-21T02:05:54.799262+0000","last_clean":"2024-09-21T02:05:54.799262+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:54.799262+0000","last_undegraded":"2024-09-21T02:05:54.799262+0000","last_fullsized":"2024-09-21T02:05:54.799262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752531+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:05:54.752531+0000","last_peered":"2024-09-21T02:05:54.752531+0000","last_clean":"2024-09-21T02:05:54.752531+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:05:54.752531+0000","last_undegraded":"2024-09-21T02:05:54.752531+0000","last_fullsized":"2024-09-21T02:05:54.752531+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694014+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:05:54.694014+0000","last_peered":"2024-09-21T02:05:54.694014+0000","last_clean":"2024-09-21T02:05:54.694014+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:05:54.694014+0000","last_undegraded":"2024-09-21T02:05:54.694014+0000","last_fullsized":"2024-09-21T02:05:54.694014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314403+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:54.314403+0000","last_peered":"2024-09-21T02:05:54.314403+0000","last_clean":"2024-09-21T02:05:54.314403+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:54.314403+0000","last_undegraded":"2024-09-21T02:05:54.314403+0000","last_fullsized":"2024-09-21T02:05:54.314403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799508+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:05:54.799508+0000","last_peered":"2024-09-21T02:05:54.799508+0000","last_clean":"2024-09-21T02:05:54.799508+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:05:54.799508+0000","last_undegraded":"2024-09-21T02:05:54.799508+0000","last_fullsized":"2024-09-21T02:05:54.799508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799481+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:05:54.799481+0000","last_peered":"2024-09-21T02:05:54.799481+0000","last_clean":"2024-09-21T02:05:54.799481+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:05:54.799481+0000","last_undegraded":"2024-09-21T02:05:54.799481+0000","last_fullsized":"2024-09-21T02:05:54.799481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799792+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:05:54.799792+0000","last_peered":"2024-09-21T02:05:54.799792+0000","last_clean":"2024-09-21T02:05:54.799792+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:05:54.799792+0000","last_undegraded":"2024-09-21T02:05:54.799792+0000","last_fullsized":"2024-09-21T02:05:54.799792+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"84'6","reported_seq":99,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:09.819153+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:06:09.819153+0000","last_peered":"2024-09-21T02:06:09.819153+0000","last_clean":"2024-09-21T02:06:09.819153+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:06:09.819153+0000","last_undegraded":"2024-09-21T02:06:09.819153+0000","last_fullsized":"2024-09-21T02:06:09.819153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838886+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:05:54.838886+0000","last_peered":"2024-09-21T02:05:54.838886+0000","last_clean":"2024-09-21T02:05:54.838886+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:05:54.838886+0000","last_undegraded":"2024-09-21T02:05:54.838886+0000","last_fullsized":"2024-09-21T02:05:54.838886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799605+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:05:54.799605+0000","last_peered":"2024-09-21T02:05:54.799605+0000","last_clean":"2024-09-21T02:05:54.799605+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:05:54.799605+0000","last_undegraded":"2024-09-21T02:05:54.799605+0000","last_fullsized":"2024-09-21T02:05:54.799605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752994+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:05:54.752994+0000","last_peered":"2024-09-21T02:05:54.752994+0000","last_clean":"2024-09-21T02:05:54.752994+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:05:54.752994+0000","last_undegraded":"2024-09-21T02:05:54.752994+0000","last_fullsized":"2024-09-21T02:05:54.752994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838525+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:05:54.838525+0000","last_peered":"2024-09-21T02:05:54.838525+0000","last_clean":"2024-09-21T02:05:54.838525+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:05:54.838525+0000","last_undegraded":"2024-09-21T02:05:54.838525+0000","last_fullsized":"2024-09-21T02:05:54.838525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799764+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:05:54.799764+0000","last_peered":"2024-09-21T02:05:54.799764+0000","last_clean":"2024-09-21T02:05:54.799764+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:05:54.799764+0000","last_undegraded":"2024-09-21T02:05:54.799764+0000","last_fullsized":"2024-09-21T02:05:54.799764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309190+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:54.309190+0000","last_peered":"2024-09-21T02:05:54.309190+0000","last_clean":"2024-09-21T02:05:54.309190+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:54.309190+0000","last_undegraded":"2024-09-21T02:05:54.309190+0000","last_fullsized":"2024-09-21T02:05:54.309190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751681+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:05:54.751681+0000","last_peered":"2024-09-21T02:05:54.751681+0000","last_clean":"2024-09-21T02:05:54.751681+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:05:54.751681+0000","last_undegraded":"2024-09-21T02:05:54.751681+0000","last_fullsized":"2024-09-21T02:05:54.751681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314237+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:05:54.314237+0000","last_peered":"2024-09-21T02:05:54.314237+0000","last_clean":"2024-09-21T02:05:54.314237+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:05:54.314237+0000","last_undegraded":"2024-09-21T02:05:54.314237+0000","last_fullsized":"2024-09-21T02:05:54.314237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840344+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:05:54.840344+0000","last_peered":"2024-09-21T02:05:54.840344+0000","last_clean":"2024-09-21T02:05:54.840344+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:05:54.840344+0000","last_undegraded":"2024-09-21T02:05:54.840344+0000","last_fullsized":"2024-09-21T02:05:54.840344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751927+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:05:54.751927+0000","last_peered":"2024-09-21T02:05:54.751927+0000","last_clean":"2024-09-21T02:05:54.751927+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:05:54.751927+0000","last_undegraded":"2024-09-21T02:05:54.751927+0000","last_fullsized":"2024-09-21T02:05:54.751927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752267+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:05:54.752267+0000","last_peered":"2024-09-21T02:05:54.752267+0000","last_clean":"2024-09-21T02:05:54.752267+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:05:54.752267+0000","last_undegraded":"2024-09-21T02:05:54.752267+0000","last_fullsized":"2024-09-21T02:05:54.752267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839541+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:54.839541+0000","last_peered":"2024-09-21T02:05:54.839541+0000","last_clean":"2024-09-21T02:05:54.839541+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:54.839541+0000","last_undegraded":"2024-09-21T02:05:54.839541+0000","last_fullsized":"2024-09-21T02:05:54.839541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309164+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:05:54.309164+0000","last_peered":"2024-09-21T02:05:54.309164+0000","last_clean":"2024-09-21T02:05:54.309164+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:05:54.309164+0000","last_undegraded":"2024-09-21T02:05:54.309164+0000","last_fullsized":"2024-09-21T02:05:54.309164+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799119+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:05:54.799119+0000","last_peered":"2024-09-21T02:05:54.799119+0000","last_clean":"2024-09-21T02:05:54.799119+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:05:54.799119+0000","last_undegraded":"2024-09-21T02:05:54.799119+0000","last_fullsized":"2024-09-21T02:05:54.799119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":101,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:11.821867+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:06:11.821867+0000","last_peered":"2024-09-21T02:06:11.821867+0000","last_clean":"2024-09-21T02:06:11.821867+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:06:11.821867+0000","last_undegraded":"2024-09-21T02:06:11.821867+0000","last_fullsized":"2024-09-21T02:06:11.821867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796362+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:05:54.796362+0000","last_peered":"2024-09-21T02:05:54.796362+0000","last_clean":"2024-09-21T02:05:54.796362+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:05:54.796362+0000","last_undegraded":"2024-09-21T02:05:54.796362+0000","last_fullsized":"2024-09-21T02:05:54.796362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799339+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:05:54.799339+0000","last_peered":"2024-09-21T02:05:54.799339+0000","last_clean":"2024-09-21T02:05:54.799339+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:05:54.799339+0000","last_undegraded":"2024-09-21T02:05:54.799339+0000","last_fullsized":"2024-09-21T02:05:54.799339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752282+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:54.752282+0000","last_peered":"2024-09-21T02:05:54.752282+0000","last_clean":"2024-09-21T02:05:54.752282+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:54.752282+0000","last_undegraded":"2024-09-21T02:05:54.752282+0000","last_fullsized":"2024-09-21T02:05:54.752282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314270+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:05:54.314270+0000","last_peered":"2024-09-21T02:05:54.314270+0000","last_clean":"2024-09-21T02:05:54.314270+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:05:54.314270+0000","last_undegraded":"2024-09-21T02:05:54.314270+0000","last_fullsized":"2024-09-21T02:05:54.314270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308650+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:05:54.308650+0000","last_peered":"2024-09-21T02:05:54.308650+0000","last_clean":"2024-09-21T02:05:54.308650+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:05:54.308650+0000","last_undegraded":"2024-09-21T02:05:54.308650+0000","last_fullsized":"2024-09-21T02:05:54.308650+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309809+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:54.309809+0000","last_peered":"2024-09-21T02:05:54.309809+0000","last_clean":"2024-09-21T02:05:54.309809+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:54.309809+0000","last_undegraded":"2024-09-21T02:05:54.309809+0000","last_fullsized":"2024-09-21T02:05:54.309809+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752687+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:05:54.752687+0000","last_peered":"2024-09-21T02:05:54.752687+0000","last_clean":"2024-09-21T02:05:54.752687+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:05:54.752687+0000","last_undegraded":"2024-09-21T02:05:54.752687+0000","last_fullsized":"2024-09-21T02:05:54.752687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840081+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:05:54.840081+0000","last_peered":"2024-09-21T02:05:54.840081+0000","last_clean":"2024-09-21T02:05:54.840081+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:05:54.840081+0000","last_undegraded":"2024-09-21T02:05:54.840081+0000","last_fullsized":"2024-09-21T02:05:54.840081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'34","reported_seq":188,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.835935+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:06:12.835935+0000","last_peered":"2024-09-21T02:06:12.835935+0000","last_clean":"2024-09-21T02:06:12.835935+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:06:12.835935+0000","last_undegraded":"2024-09-21T02:06:12.835935+0000","last_fullsized":"2024-09-21T02:06:12.835935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":122,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795664+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:05:54.795664+0000","last_peered":"2024-09-21T02:05:54.795664+0000","last_clean":"2024-09-21T02:05:54.795664+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:05:54.795664+0000","last_undegraded":"2024-09-21T02:05:54.795664+0000","last_fullsized":"2024-09-21T02:05:54.795664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839159+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:05:54.839159+0000","last_peered":"2024-09-21T02:05:54.839159+0000","last_clean":"2024-09-21T02:05:54.839159+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:05:54.839159+0000","last_undegraded":"2024-09-21T02:05:54.839159+0000","last_fullsized":"2024-09-21T02:05:54.839159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752902+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:54.752902+0000","last_peered":"2024-09-21T02:05:54.752902+0000","last_clean":"2024-09-21T02:05:54.752902+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:54.752902+0000","last_undegraded":"2024-09-21T02:05:54.752902+0000","last_fullsized":"2024-09-21T02:05:54.752902+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314110+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:05:54.314110+0000","last_peered":"2024-09-21T02:05:54.314110+0000","last_clean":"2024-09-21T02:05:54.314110+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:05:54.314110+0000","last_undegraded":"2024-09-21T02:05:54.314110+0000","last_fullsized":"2024-09-21T02:05:54.314110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838908+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:05:54.838908+0000","last_peered":"2024-09-21T02:05:54.838908+0000","last_clean":"2024-09-21T02:05:54.838908+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:05:54.838908+0000","last_undegraded":"2024-09-21T02:05:54.838908+0000","last_fullsized":"2024-09-21T02:05:54.838908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799598+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:54.799598+0000","last_peered":"2024-09-21T02:05:54.799598+0000","last_clean":"2024-09-21T02:05:54.799598+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:54.799598+0000","last_undegraded":"2024-09-21T02:05:54.799598+0000","last_fullsized":"2024-09-21T02:05:54.799598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839390+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:05:54.839390+0000","last_peered":"2024-09-21T02:05:54.839390+0000","last_clean":"2024-09-21T02:05:54.839390+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:05:54.839390+0000","last_undegraded":"2024-09-21T02:05:54.839390+0000","last_fullsized":"2024-09-21T02:05:54.839390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839423+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:05:54.839423+0000","last_peered":"2024-09-21T02:05:54.839423+0000","last_clean":"2024-09-21T02:05:54.839423+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:05:54.839423+0000","last_undegraded":"2024-09-21T02:05:54.839423+0000","last_fullsized":"2024-09-21T02:05:54.839423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":149,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.822897+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:06:12.822897+0000","last_peered":"2024-09-21T02:06:12.822897+0000","last_clean":"2024-09-21T02:06:12.822897+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:06:12.822897+0000","last_undegraded":"2024-09-21T02:06:12.822897+0000","last_fullsized":"2024-09-21T02:06:12.822897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693584+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:05:54.693584+0000","last_peered":"2024-09-21T02:05:54.693584+0000","last_clean":"2024-09-21T02:05:54.693584+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:05:54.693584+0000","last_undegraded":"2024-09-21T02:05:54.693584+0000","last_fullsized":"2024-09-21T02:05:54.693584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838626+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:05:54.838626+0000","last_peered":"2024-09-21T02:05:54.838626+0000","last_clean":"2024-09-21T02:05:54.838626+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:05:54.838626+0000","last_undegraded":"2024-09-21T02:05:54.838626+0000","last_fullsized":"2024-09-21T02:05:54.838626+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795480+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:05:54.795480+0000","last_peered":"2024-09-21T02:05:54.795480+0000","last_clean":"2024-09-21T02:05:54.795480+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:05:54.795480+0000","last_undegraded":"2024-09-21T02:05:54.795480+0000","last_fullsized":"2024-09-21T02:05:54.795480+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752188+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:54.752188+0000","last_peered":"2024-09-21T02:05:54.752188+0000","last_clean":"2024-09-21T02:05:54.752188+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:54.752188+0000","last_undegraded":"2024-09-21T02:05:54.752188+0000","last_fullsized":"2024-09-21T02:05:54.752188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841549+0000","last_change":"2024-09-21T02:05:54.841426+0000","last_active":"2024-09-21T02:05:54.841549+0000","last_peered":"2024-09-21T02:05:54.841549+0000","last_clean":"2024-09-21T02:05:54.841549+0000","last_became_active":"2024-09-21T02:05:54.840855+0000","last_became_peered":"2024-09-21T02:05:54.840855+0000","last_unstale":"2024-09-21T02:05:54.841549+0000","last_undegraded":"2024-09-21T02:05:54.841549+0000","last_fullsized":"2024-09-21T02:05:54.841549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:40.637427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693707+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:05:54.693707+0000","last_peered":"2024-09-21T02:05:54.693707+0000","last_clean":"2024-09-21T02:05:54.693707+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:05:54.693707+0000","last_undegraded":"2024-09-21T02:05:54.693707+0000","last_fullsized":"2024-09-21T02:05:54.693707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308544+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:54.308544+0000","last_peered":"2024-09-21T02:05:54.308544+0000","last_clean":"2024-09-21T02:05:54.308544+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:54.308544+0000","last_undegraded":"2024-09-21T02:05:54.308544+0000","last_fullsized":"2024-09-21T02:05:54.308544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309450+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:54.309450+0000","last_peered":"2024-09-21T02:05:54.309450+0000","last_clean":"2024-09-21T02:05:54.309450+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:54.309450+0000","last_undegraded":"2024-09-21T02:05:54.309450+0000","last_fullsized":"2024-09-21T02:05:54.309450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752299+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:05:54.752299+0000","last_peered":"2024-09-21T02:05:54.752299+0000","last_clean":"2024-09-21T02:05:54.752299+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:05:54.752299+0000","last_undegraded":"2024-09-21T02:05:54.752299+0000","last_fullsized":"2024-09-21T02:05:54.752299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309752+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:05:54.309752+0000","last_peered":"2024-09-21T02:05:54.309752+0000","last_clean":"2024-09-21T02:05:54.309752+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:05:54.309752+0000","last_undegraded":"2024-09-21T02:05:54.309752+0000","last_fullsized":"2024-09-21T02:05:54.309752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'6","reported_seq":102,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.823811+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:06:12.823811+0000","last_peered":"2024-09-21T02:06:12.823811+0000","last_clean":"2024-09-21T02:06:12.823811+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:06:12.823811+0000","last_undegraded":"2024-09-21T02:06:12.823811+0000","last_fullsized":"2024-09-21T02:06:12.823811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839609+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:05:54.839609+0000","last_peered":"2024-09-21T02:05:54.839609+0000","last_clean":"2024-09-21T02:05:54.839609+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:05:54.839609+0000","last_undegraded":"2024-09-21T02:05:54.839609+0000","last_fullsized":"2024-09-21T02:05:54.839609+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309832+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:05:54.309832+0000","last_peered":"2024-09-21T02:05:54.309832+0000","last_clean":"2024-09-21T02:05:54.309832+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:05:54.309832+0000","last_undegraded":"2024-09-21T02:05:54.309832+0000","last_fullsized":"2024-09-21T02:05:54.309832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313822+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:05:54.313822+0000","last_peered":"2024-09-21T02:05:54.313822+0000","last_clean":"2024-09-21T02:05:54.313822+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:05:54.313822+0000","last_undegraded":"2024-09-21T02:05:54.313822+0000","last_fullsized":"2024-09-21T02:05:54.313822+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309878+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:05:54.309878+0000","last_peered":"2024-09-21T02:05:54.309878+0000","last_clean":"2024-09-21T02:05:54.309878+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:05:54.309878+0000","last_undegraded":"2024-09-21T02:05:54.309878+0000","last_fullsized":"2024-09-21T02:05:54.309878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309595+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:05:54.309595+0000","last_peered":"2024-09-21T02:05:54.309595+0000","last_clean":"2024-09-21T02:05:54.309595+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:05:54.309595+0000","last_undegraded":"2024-09-21T02:05:54.309595+0000","last_fullsized":"2024-09-21T02:05:54.309595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838375+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:05:54.838375+0000","last_peered":"2024-09-21T02:05:54.838375+0000","last_clean":"2024-09-21T02:05:54.838375+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:05:54.838375+0000","last_undegraded":"2024-09-21T02:05:54.838375+0000","last_fullsized":"2024-09-21T02:05:54.838375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308869+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:05:54.308869+0000","last_peered":"2024-09-21T02:05:54.308869+0000","last_clean":"2024-09-21T02:05:54.308869+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:05:54.308869+0000","last_undegraded":"2024-09-21T02:05:54.308869+0000","last_fullsized":"2024-09-21T02:05:54.308869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314470+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:05:54.314470+0000","last_peered":"2024-09-21T02:05:54.314470+0000","last_clean":"2024-09-21T02:05:54.314470+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:05:54.314470+0000","last_undegraded":"2024-09-21T02:05:54.314470+0000","last_fullsized":"2024-09-21T02:05:54.314470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310118+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:05:54.310118+0000","last_peered":"2024-09-21T02:05:54.310118+0000","last_clean":"2024-09-21T02:05:54.310118+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:05:54.310118+0000","last_undegraded":"2024-09-21T02:05:54.310118+0000","last_fullsized":"2024-09-21T02:05:54.310118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694096+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:05:54.694096+0000","last_peered":"2024-09-21T02:05:54.694096+0000","last_clean":"2024-09-21T02:05:54.694096+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:05:54.694096+0000","last_undegraded":"2024-09-21T02:05:54.694096+0000","last_fullsized":"2024-09-21T02:05:54.694096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751904+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:05:54.751904+0000","last_peered":"2024-09-21T02:05:54.751904+0000","last_clean":"2024-09-21T02:05:54.751904+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:05:54.751904+0000","last_undegraded":"2024-09-21T02:05:54.751904+0000","last_fullsized":"2024-09-21T02:05:54.751904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693654+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:05:54.693654+0000","last_peered":"2024-09-21T02:05:54.693654+0000","last_clean":"2024-09-21T02:05:54.693654+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:05:54.693654+0000","last_undegraded":"2024-09-21T02:05:54.693654+0000","last_fullsized":"2024-09-21T02:05:54.693654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839095+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:05:54.839095+0000","last_peered":"2024-09-21T02:05:54.839095+0000","last_clean":"2024-09-21T02:05:54.839095+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:05:54.839095+0000","last_undegraded":"2024-09-21T02:05:54.839095+0000","last_fullsized":"2024-09-21T02:05:54.839095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308899+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:05:54.308899+0000","last_peered":"2024-09-21T02:05:54.308899+0000","last_clean":"2024-09-21T02:05:54.308899+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:05:54.308899+0000","last_undegraded":"2024-09-21T02:05:54.308899+0000","last_fullsized":"2024-09-21T02:05:54.308899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314056+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:05:54.314056+0000","last_peered":"2024-09-21T02:05:54.314056+0000","last_clean":"2024-09-21T02:05:54.314056+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:05:54.314056+0000","last_undegraded":"2024-09-21T02:05:54.314056+0000","last_fullsized":"2024-09-21T02:05:54.314056+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310008+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:05:54.310008+0000","last_peered":"2024-09-21T02:05:54.310008+0000","last_clean":"2024-09-21T02:05:54.310008+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:05:54.310008+0000","last_undegraded":"2024-09-21T02:05:54.310008+0000","last_fullsized":"2024-09-21T02:05:54.310008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796157+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:05:54.796157+0000","last_peered":"2024-09-21T02:05:54.796157+0000","last_clean":"2024-09-21T02:05:54.796157+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:05:54.796157+0000","last_undegraded":"2024-09-21T02:05:54.796157+0000","last_fullsized":"2024-09-21T02:05:54.796157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314535+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:05:54.314535+0000","last_peered":"2024-09-21T02:05:54.314535+0000","last_clean":"2024-09-21T02:05:54.314535+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:05:54.314535+0000","last_undegraded":"2024-09-21T02:05:54.314535+0000","last_fullsized":"2024-09-21T02:05:54.314535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838862+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:05:54.838862+0000","last_peered":"2024-09-21T02:05:54.838862+0000","last_clean":"2024-09-21T02:05:54.838862+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:05:54.838862+0000","last_undegraded":"2024-09-21T02:05:54.838862+0000","last_fullsized":"2024-09-21T02:05:54.838862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838248+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:05:54.838248+0000","last_peered":"2024-09-21T02:05:54.838248+0000","last_clean":"2024-09-21T02:05:54.838248+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:05:54.838248+0000","last_undegraded":"2024-09-21T02:05:54.838248+0000","last_fullsized":"2024-09-21T02:05:54.838248+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838241+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:05:54.838241+0000","last_peered":"2024-09-21T02:05:54.838241+0000","last_clean":"2024-09-21T02:05:54.838241+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:05:54.838241+0000","last_undegraded":"2024-09-21T02:05:54.838241+0000","last_fullsized":"2024-09-21T02:05:54.838241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308871+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:05:54.308871+0000","last_peered":"2024-09-21T02:05:54.308871+0000","last_clean":"2024-09-21T02:05:54.308871+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:05:54.308871+0000","last_undegraded":"2024-09-21T02:05:54.308871+0000","last_fullsized":"2024-09-21T02:05:54.308871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314437+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:05:54.314437+0000","last_peered":"2024-09-21T02:05:54.314437+0000","last_clean":"2024-09-21T02:05:54.314437+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:05:54.314437+0000","last_undegraded":"2024-09-21T02:05:54.314437+0000","last_fullsized":"2024-09-21T02:05:54.314437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799721+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:05:54.799721+0000","last_peered":"2024-09-21T02:05:54.799721+0000","last_clean":"2024-09-21T02:05:54.799721+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:05:54.799721+0000","last_undegraded":"2024-09-21T02:05:54.799721+0000","last_fullsized":"2024-09-21T02:05:54.799721+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310103+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:05:54.310103+0000","last_peered":"2024-09-21T02:05:54.310103+0000","last_clean":"2024-09-21T02:05:54.310103+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:05:54.310103+0000","last_undegraded":"2024-09-21T02:05:54.310103+0000","last_fullsized":"2024-09-21T02:05:54.310103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839470+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:05:54.839470+0000","last_peered":"2024-09-21T02:05:54.839470+0000","last_clean":"2024-09-21T02:05:54.839470+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:05:54.839470+0000","last_undegraded":"2024-09-21T02:05:54.839470+0000","last_fullsized":"2024-09-21T02:05:54.839470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693644+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:05:54.693644+0000","last_peered":"2024-09-21T02:05:54.693644+0000","last_clean":"2024-09-21T02:05:54.693644+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:05:54.693644+0000","last_undegraded":"2024-09-21T02:05:54.693644+0000","last_fullsized":"2024-09-21T02:05:54.693644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799161+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:05:54.799161+0000","last_peered":"2024-09-21T02:05:54.799161+0000","last_clean":"2024-09-21T02:05:54.799161+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:05:54.799161+0000","last_undegraded":"2024-09-21T02:05:54.799161+0000","last_fullsized":"2024-09-21T02:05:54.799161+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309453+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:05:54.309453+0000","last_peered":"2024-09-21T02:05:54.309453+0000","last_clean":"2024-09-21T02:05:54.309453+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:05:54.309453+0000","last_undegraded":"2024-09-21T02:05:54.309453+0000","last_fullsized":"2024-09-21T02:05:54.309453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839817+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:05:54.839817+0000","last_peered":"2024-09-21T02:05:54.839817+0000","last_clean":"2024-09-21T02:05:54.839817+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:05:54.839817+0000","last_undegraded":"2024-09-21T02:05:54.839817+0000","last_fullsized":"2024-09-21T02:05:54.839817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314480+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:05:54.314480+0000","last_peered":"2024-09-21T02:05:54.314480+0000","last_clean":"2024-09-21T02:05:54.314480+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:05:54.314480+0000","last_undegraded":"2024-09-21T02:05:54.314480+0000","last_fullsized":"2024-09-21T02:05:54.314480+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838525+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:05:54.838525+0000","last_peered":"2024-09-21T02:05:54.838525+0000","last_clean":"2024-09-21T02:05:54.838525+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:05:54.838525+0000","last_undegraded":"2024-09-21T02:05:54.838525+0000","last_fullsized":"2024-09-21T02:05:54.838525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838614+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:05:54.838614+0000","last_peered":"2024-09-21T02:05:54.838614+0000","last_clean":"2024-09-21T02:05:54.838614+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:05:54.838614+0000","last_undegraded":"2024-09-21T02:05:54.838614+0000","last_fullsized":"2024-09-21T02:05:54.838614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314403+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:05:54.314403+0000","last_peered":"2024-09-21T02:05:54.314403+0000","last_clean":"2024-09-21T02:05:54.314403+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:05:54.314403+0000","last_undegraded":"2024-09-21T02:05:54.314403+0000","last_fullsized":"2024-09-21T02:05:54.314403+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74198,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":576,"num_read_kb":369,"num_write":251,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":222594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710754,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":500919,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874278,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1105920,"data_stored":1153835,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037804,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":504442,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":500612,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397558,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1024000,"data_stored":1098003,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561084,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":950272,"data_stored":496241,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1032192,"data_stored":1092803,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":958464,"data_stored":500315,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":962560,"data_stored":495558,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378704,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1036288,"data_stored":563982,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1032192,"data_stored":563595,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":974848,"data_stored":497697,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:06:18.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:06:18.167 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:17 smithi161 ceph-mon[34061]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:17 smithi123 ceph-mon[33453]: pgmap v326: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:18.427 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:18.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:18 smithi016 ceph-mon[31245]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 -- 172.21.15.16:0/2409756941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 msgr2=0x7f0ac41087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:18.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 --2- 172.21.15.16:0/2409756941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac41087b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009900 tx=0x7f0aac02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 -- 172.21.15.16:0/2409756941 shutdown_connections 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 --2- 172.21.15.16:0/2409756941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 --2- 172.21.15.16:0/2409756941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac41087b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 --2- 172.21.15.16:0/2409756941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 0x7f0ac4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 -- 172.21.15.16:0/2409756941 >> 172.21.15.16:0/2409756941 conn(0x7f0ac4079730 msgr2=0x7f0ac4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 -- 172.21.15.16:0/2409756941 shutdown_connections 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.926+0000 7f0acac19640 1 -- 172.21.15.16:0/2409756941 wait complete. 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 Processor -- start 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 -- start start 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 0x7f0ac419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:18.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4113930 con 0x7f0ac41083a0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac41137b0 con 0x7f0ac4102a60 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0acac19640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4113ab0 con 0x7f0ac4108cf0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0ac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.927+0000 7f0ac3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38846/0 (socket says 172.21.15.16:38846) 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 -- 172.21.15.16:0/606429509 learned_addr learned my addr 172.21.15.16:0/606429509 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac89b5640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 0x7f0ac419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac91b6640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 -- 172.21.15.16:0/606429509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 msgr2=0x7f0ac419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 -- 172.21.15.16:0/606429509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 msgr2=0x7f0ac419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 0x7f0ac419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 -- 172.21.15.16:0/606429509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac41a0310 con 0x7f0ac41083a0 2024-09-21T02:06:18.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac91b6640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac3fff640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0aac005940 tx=0x7f0aac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac03f030 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac41a05a0 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aac003c90 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.928+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac035070 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.929+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac41a6d60 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.930+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac4102e70 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.931+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0aac042030 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.932+0000 7f0ac1ffb640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 0x7f0ab00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.932+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0aac1082f0 con 0x7f0ac41083a0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.932+0000 7f0ac89b5640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 0x7f0ab00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:18.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.933+0000 7f0ac89b5640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 0x7f0ab00b8b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400bbe0 tx=0x7f0ab400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:18.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:18.935+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0aac0ce590 con 0x7f0ac41083a0 2024-09-21T02:06:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.121+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ac4063800 con 0x7f0ab00b65f0 2024-09-21T02:06:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.127+0000 7f0ac1ffb640 1 -- 172.21.15.16:0/606429509 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340649 (secure 0 0 0) 0x7f0ac4063800 con 0x7f0ab00b65f0 2024-09-21T02:06:19.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:06:19.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 msgr2=0x7f0ab00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 0x7f0ab00b8b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400bbe0 tx=0x7f0ab400b390 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 msgr2=0x7f0ac419a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0aac005940 tx=0x7f0aac0042f0 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 shutdown_connections 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0ab00b65f0 0x7f0ab00b8b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ac4108cf0 0x7f0ac419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ac41083a0 0x7f0ac419a940 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 --2- 172.21.15.16:0/606429509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac4102a60 0x7f0ac419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 >> 172.21.15.16:0/606429509 conn(0x7f0ac4079730 msgr2=0x7f0ac41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 shutdown_connections 2024-09-21T02:06:19.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:19.131+0000 7f0acac19640 1 -- 172.21.15.16:0/606429509 wait complete. 2024-09-21T02:06:19.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:18 smithi161 ceph-mon[34061]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:18 smithi123 ceph-mon[33453]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:19.974 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-21T02:06:18.044609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664800,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":667,"num_read_kb":443,"num_write":376,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394292,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547532,"statfs":{"total":1151940427776,"available":1151536672768,"internally_reserved":0,"allocated":11976704,"data_stored":7968002,"data_compressed":465210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003828"},"pg_stats":[{"pgid":"2.1f","version":"84'13","reported_seq":110,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:15.826307+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:06:15.826307+0000","last_peered":"2024-09-21T02:06:15.826307+0000","last_clean":"2024-09-21T02:06:15.826307+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:06:15.826307+0000","last_undegraded":"2024-09-21T02:06:15.826307+0000","last_fullsized":"2024-09-21T02:06:15.826307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693911+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:05:54.693911+0000","last_peered":"2024-09-21T02:05:54.693911+0000","last_clean":"2024-09-21T02:05:54.693911+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:05:54.693911+0000","last_undegraded":"2024-09-21T02:05:54.693911+0000","last_fullsized":"2024-09-21T02:05:54.693911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309473+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:05:54.309473+0000","last_peered":"2024-09-21T02:05:54.309473+0000","last_clean":"2024-09-21T02:05:54.309473+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:05:54.309473+0000","last_undegraded":"2024-09-21T02:05:54.309473+0000","last_fullsized":"2024-09-21T02:05:54.309473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694295+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:05:54.694295+0000","last_peered":"2024-09-21T02:05:54.694295+0000","last_clean":"2024-09-21T02:05:54.694295+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:05:54.694295+0000","last_undegraded":"2024-09-21T02:05:54.694295+0000","last_fullsized":"2024-09-21T02:05:54.694295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838888+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:05:54.838888+0000","last_peered":"2024-09-21T02:05:54.838888+0000","last_clean":"2024-09-21T02:05:54.838888+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:05:54.838888+0000","last_undegraded":"2024-09-21T02:05:54.838888+0000","last_fullsized":"2024-09-21T02:05:54.838888+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313792+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:05:54.313792+0000","last_peered":"2024-09-21T02:05:54.313792+0000","last_clean":"2024-09-21T02:05:54.313792+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:05:54.313792+0000","last_undegraded":"2024-09-21T02:05:54.313792+0000","last_fullsized":"2024-09-21T02:05:54.313792+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314180+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:54.314180+0000","last_peered":"2024-09-21T02:05:54.314180+0000","last_clean":"2024-09-21T02:05:54.314180+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:54.314180+0000","last_undegraded":"2024-09-21T02:05:54.314180+0000","last_fullsized":"2024-09-21T02:05:54.314180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839131+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:05:54.839131+0000","last_peered":"2024-09-21T02:05:54.839131+0000","last_clean":"2024-09-21T02:05:54.839131+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:05:54.839131+0000","last_undegraded":"2024-09-21T02:05:54.839131+0000","last_fullsized":"2024-09-21T02:05:54.839131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841186+0000","last_change":"2024-09-21T02:05:54.841083+0000","last_active":"2024-09-21T02:05:54.841186+0000","last_peered":"2024-09-21T02:05:54.841186+0000","last_clean":"2024-09-21T02:05:54.841186+0000","last_became_active":"2024-09-21T02:05:54.840645+0000","last_became_peered":"2024-09-21T02:05:54.840645+0000","last_unstale":"2024-09-21T02:05:54.841186+0000","last_undegraded":"2024-09-21T02:05:54.841186+0000","last_fullsized":"2024-09-21T02:05:54.841186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:33:35.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840245+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:54.840245+0000","last_peered":"2024-09-21T02:05:54.840245+0000","last_clean":"2024-09-21T02:05:54.840245+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:54.840245+0000","last_undegraded":"2024-09-21T02:05:54.840245+0000","last_fullsized":"2024-09-21T02:05:54.840245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795801+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:05:54.795801+0000","last_peered":"2024-09-21T02:05:54.795801+0000","last_clean":"2024-09-21T02:05:54.795801+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:05:54.795801+0000","last_undegraded":"2024-09-21T02:05:54.795801+0000","last_fullsized":"2024-09-21T02:05:54.795801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313927+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:05:54.313927+0000","last_peered":"2024-09-21T02:05:54.313927+0000","last_clean":"2024-09-21T02:05:54.313927+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:05:54.313927+0000","last_undegraded":"2024-09-21T02:05:54.313927+0000","last_fullsized":"2024-09-21T02:05:54.313927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799300+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:05:54.799300+0000","last_peered":"2024-09-21T02:05:54.799300+0000","last_clean":"2024-09-21T02:05:54.799300+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:05:54.799300+0000","last_undegraded":"2024-09-21T02:05:54.799300+0000","last_fullsized":"2024-09-21T02:05:54.799300+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839857+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:54.839857+0000","last_peered":"2024-09-21T02:05:54.839857+0000","last_clean":"2024-09-21T02:05:54.839857+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:54.839857+0000","last_undegraded":"2024-09-21T02:05:54.839857+0000","last_fullsized":"2024-09-21T02:05:54.839857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314162+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:05:54.314162+0000","last_peered":"2024-09-21T02:05:54.314162+0000","last_clean":"2024-09-21T02:05:54.314162+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:05:54.314162+0000","last_undegraded":"2024-09-21T02:05:54.314162+0000","last_fullsized":"2024-09-21T02:05:54.314162+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693857+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:05:54.693857+0000","last_peered":"2024-09-21T02:05:54.693857+0000","last_clean":"2024-09-21T02:05:54.693857+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:05:54.693857+0000","last_undegraded":"2024-09-21T02:05:54.693857+0000","last_fullsized":"2024-09-21T02:05:54.693857+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309822+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:05:54.309822+0000","last_peered":"2024-09-21T02:05:54.309822+0000","last_clean":"2024-09-21T02:05:54.309822+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:05:54.309822+0000","last_undegraded":"2024-09-21T02:05:54.309822+0000","last_fullsized":"2024-09-21T02:05:54.309822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314166+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:05:54.314166+0000","last_peered":"2024-09-21T02:05:54.314166+0000","last_clean":"2024-09-21T02:05:54.314166+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:05:54.314166+0000","last_undegraded":"2024-09-21T02:05:54.314166+0000","last_fullsized":"2024-09-21T02:05:54.314166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313914+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:05:54.313914+0000","last_peered":"2024-09-21T02:05:54.313914+0000","last_clean":"2024-09-21T02:05:54.313914+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:05:54.313914+0000","last_undegraded":"2024-09-21T02:05:54.313914+0000","last_fullsized":"2024-09-21T02:05:54.313914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693836+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:54.693836+0000","last_peered":"2024-09-21T02:05:54.693836+0000","last_clean":"2024-09-21T02:05:54.693836+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:54.693836+0000","last_undegraded":"2024-09-21T02:05:54.693836+0000","last_fullsized":"2024-09-21T02:05:54.693836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.754062+0000","last_change":"2024-09-21T02:05:54.753929+0000","last_active":"2024-09-21T02:05:54.754062+0000","last_peered":"2024-09-21T02:05:54.754062+0000","last_clean":"2024-09-21T02:05:54.754062+0000","last_became_active":"2024-09-21T02:05:54.753431+0000","last_became_peered":"2024-09-21T02:05:54.753431+0000","last_unstale":"2024-09-21T02:05:54.754062+0000","last_undegraded":"2024-09-21T02:05:54.754062+0000","last_fullsized":"2024-09-21T02:05:54.754062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:20:41.311972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839451+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:05:54.839451+0000","last_peered":"2024-09-21T02:05:54.839451+0000","last_clean":"2024-09-21T02:05:54.839451+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:05:54.839451+0000","last_undegraded":"2024-09-21T02:05:54.839451+0000","last_fullsized":"2024-09-21T02:05:54.839451+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839025+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:54.839025+0000","last_peered":"2024-09-21T02:05:54.839025+0000","last_clean":"2024-09-21T02:05:54.839025+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:54.839025+0000","last_undegraded":"2024-09-21T02:05:54.839025+0000","last_fullsized":"2024-09-21T02:05:54.839025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314212+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:05:54.314212+0000","last_peered":"2024-09-21T02:05:54.314212+0000","last_clean":"2024-09-21T02:05:54.314212+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:05:54.314212+0000","last_undegraded":"2024-09-21T02:05:54.314212+0000","last_fullsized":"2024-09-21T02:05:54.314212+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838738+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:05:54.838738+0000","last_peered":"2024-09-21T02:05:54.838738+0000","last_clean":"2024-09-21T02:05:54.838738+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:05:54.838738+0000","last_undegraded":"2024-09-21T02:05:54.838738+0000","last_fullsized":"2024-09-21T02:05:54.838738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314577+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:54.314577+0000","last_peered":"2024-09-21T02:05:54.314577+0000","last_clean":"2024-09-21T02:05:54.314577+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:54.314577+0000","last_undegraded":"2024-09-21T02:05:54.314577+0000","last_fullsized":"2024-09-21T02:05:54.314577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309620+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:05:54.309620+0000","last_peered":"2024-09-21T02:05:54.309620+0000","last_clean":"2024-09-21T02:05:54.309620+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:05:54.309620+0000","last_undegraded":"2024-09-21T02:05:54.309620+0000","last_fullsized":"2024-09-21T02:05:54.309620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694104+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:05:54.694104+0000","last_peered":"2024-09-21T02:05:54.694104+0000","last_clean":"2024-09-21T02:05:54.694104+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:05:54.694104+0000","last_undegraded":"2024-09-21T02:05:54.694104+0000","last_fullsized":"2024-09-21T02:05:54.694104+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'16","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796179+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:05:54.796179+0000","last_peered":"2024-09-21T02:05:54.796179+0000","last_clean":"2024-09-21T02:05:54.796179+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:05:54.796179+0000","last_undegraded":"2024-09-21T02:05:54.796179+0000","last_fullsized":"2024-09-21T02:05:54.796179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693484+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:05:54.693484+0000","last_peered":"2024-09-21T02:05:54.693484+0000","last_clean":"2024-09-21T02:05:54.693484+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:05:54.693484+0000","last_undegraded":"2024-09-21T02:05:54.693484+0000","last_fullsized":"2024-09-21T02:05:54.693484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308954+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:05:54.308954+0000","last_peered":"2024-09-21T02:05:54.308954+0000","last_clean":"2024-09-21T02:05:54.308954+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:05:54.308954+0000","last_undegraded":"2024-09-21T02:05:54.308954+0000","last_fullsized":"2024-09-21T02:05:54.308954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314094+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:54.314094+0000","last_peered":"2024-09-21T02:05:54.314094+0000","last_clean":"2024-09-21T02:05:54.314094+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:54.314094+0000","last_undegraded":"2024-09-21T02:05:54.314094+0000","last_fullsized":"2024-09-21T02:05:54.314094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313911+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:05:54.313911+0000","last_peered":"2024-09-21T02:05:54.313911+0000","last_clean":"2024-09-21T02:05:54.313911+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:05:54.313911+0000","last_undegraded":"2024-09-21T02:05:54.313911+0000","last_fullsized":"2024-09-21T02:05:54.313911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309554+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:05:54.309554+0000","last_peered":"2024-09-21T02:05:54.309554+0000","last_clean":"2024-09-21T02:05:54.309554+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:05:54.309554+0000","last_undegraded":"2024-09-21T02:05:54.309554+0000","last_fullsized":"2024-09-21T02:05:54.309554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796030+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:05:54.796030+0000","last_peered":"2024-09-21T02:05:54.796030+0000","last_clean":"2024-09-21T02:05:54.796030+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:05:54.796030+0000","last_undegraded":"2024-09-21T02:05:54.796030+0000","last_fullsized":"2024-09-21T02:05:54.796030+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839502+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:05:54.839502+0000","last_peered":"2024-09-21T02:05:54.839502+0000","last_clean":"2024-09-21T02:05:54.839502+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:05:54.839502+0000","last_undegraded":"2024-09-21T02:05:54.839502+0000","last_fullsized":"2024-09-21T02:05:54.839502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309631+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:05:54.309631+0000","last_peered":"2024-09-21T02:05:54.309631+0000","last_clean":"2024-09-21T02:05:54.309631+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:05:54.309631+0000","last_undegraded":"2024-09-21T02:05:54.309631+0000","last_fullsized":"2024-09-21T02:05:54.309631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":103,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:14.825713+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:06:14.825713+0000","last_peered":"2024-09-21T02:06:14.825713+0000","last_clean":"2024-09-21T02:06:14.825713+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:06:14.825713+0000","last_undegraded":"2024-09-21T02:06:14.825713+0000","last_fullsized":"2024-09-21T02:06:14.825713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308803+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:05:54.308803+0000","last_peered":"2024-09-21T02:05:54.308803+0000","last_clean":"2024-09-21T02:05:54.308803+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:05:54.308803+0000","last_undegraded":"2024-09-21T02:05:54.308803+0000","last_fullsized":"2024-09-21T02:05:54.308803+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752017+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:05:54.752017+0000","last_peered":"2024-09-21T02:05:54.752017+0000","last_clean":"2024-09-21T02:05:54.752017+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:05:54.752017+0000","last_undegraded":"2024-09-21T02:05:54.752017+0000","last_fullsized":"2024-09-21T02:05:54.752017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":100,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:13.824149+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:06:13.824149+0000","last_peered":"2024-09-21T02:06:13.824149+0000","last_clean":"2024-09-21T02:06:13.824149+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:06:13.824149+0000","last_undegraded":"2024-09-21T02:06:13.824149+0000","last_fullsized":"2024-09-21T02:06:13.824149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":56,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309175+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:05:54.309175+0000","last_peered":"2024-09-21T02:05:54.309175+0000","last_clean":"2024-09-21T02:05:54.309175+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:05:54.309175+0000","last_undegraded":"2024-09-21T02:05:54.309175+0000","last_fullsized":"2024-09-21T02:05:54.309175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308470+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:05:54.308470+0000","last_peered":"2024-09-21T02:05:54.308470+0000","last_clean":"2024-09-21T02:05:54.308470+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:05:54.308470+0000","last_undegraded":"2024-09-21T02:05:54.308470+0000","last_fullsized":"2024-09-21T02:05:54.308470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799262+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:54.799262+0000","last_peered":"2024-09-21T02:05:54.799262+0000","last_clean":"2024-09-21T02:05:54.799262+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:54.799262+0000","last_undegraded":"2024-09-21T02:05:54.799262+0000","last_fullsized":"2024-09-21T02:05:54.799262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752531+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:05:54.752531+0000","last_peered":"2024-09-21T02:05:54.752531+0000","last_clean":"2024-09-21T02:05:54.752531+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:05:54.752531+0000","last_undegraded":"2024-09-21T02:05:54.752531+0000","last_fullsized":"2024-09-21T02:05:54.752531+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694014+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:05:54.694014+0000","last_peered":"2024-09-21T02:05:54.694014+0000","last_clean":"2024-09-21T02:05:54.694014+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:05:54.694014+0000","last_undegraded":"2024-09-21T02:05:54.694014+0000","last_fullsized":"2024-09-21T02:05:54.694014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314403+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:54.314403+0000","last_peered":"2024-09-21T02:05:54.314403+0000","last_clean":"2024-09-21T02:05:54.314403+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:54.314403+0000","last_undegraded":"2024-09-21T02:05:54.314403+0000","last_fullsized":"2024-09-21T02:05:54.314403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799508+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:05:54.799508+0000","last_peered":"2024-09-21T02:05:54.799508+0000","last_clean":"2024-09-21T02:05:54.799508+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:05:54.799508+0000","last_undegraded":"2024-09-21T02:05:54.799508+0000","last_fullsized":"2024-09-21T02:05:54.799508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799481+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:05:54.799481+0000","last_peered":"2024-09-21T02:05:54.799481+0000","last_clean":"2024-09-21T02:05:54.799481+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:05:54.799481+0000","last_undegraded":"2024-09-21T02:05:54.799481+0000","last_fullsized":"2024-09-21T02:05:54.799481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799792+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:05:54.799792+0000","last_peered":"2024-09-21T02:05:54.799792+0000","last_clean":"2024-09-21T02:05:54.799792+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:05:54.799792+0000","last_undegraded":"2024-09-21T02:05:54.799792+0000","last_fullsized":"2024-09-21T02:05:54.799792+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"84'6","reported_seq":104,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:14.826367+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:06:14.826367+0000","last_peered":"2024-09-21T02:06:14.826367+0000","last_clean":"2024-09-21T02:06:14.826367+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:06:14.826367+0000","last_undegraded":"2024-09-21T02:06:14.826367+0000","last_fullsized":"2024-09-21T02:06:14.826367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838886+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:05:54.838886+0000","last_peered":"2024-09-21T02:05:54.838886+0000","last_clean":"2024-09-21T02:05:54.838886+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:05:54.838886+0000","last_undegraded":"2024-09-21T02:05:54.838886+0000","last_fullsized":"2024-09-21T02:05:54.838886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799605+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:05:54.799605+0000","last_peered":"2024-09-21T02:05:54.799605+0000","last_clean":"2024-09-21T02:05:54.799605+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:05:54.799605+0000","last_undegraded":"2024-09-21T02:05:54.799605+0000","last_fullsized":"2024-09-21T02:05:54.799605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752994+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:05:54.752994+0000","last_peered":"2024-09-21T02:05:54.752994+0000","last_clean":"2024-09-21T02:05:54.752994+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:05:54.752994+0000","last_undegraded":"2024-09-21T02:05:54.752994+0000","last_fullsized":"2024-09-21T02:05:54.752994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838525+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:05:54.838525+0000","last_peered":"2024-09-21T02:05:54.838525+0000","last_clean":"2024-09-21T02:05:54.838525+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:05:54.838525+0000","last_undegraded":"2024-09-21T02:05:54.838525+0000","last_fullsized":"2024-09-21T02:05:54.838525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799764+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:05:54.799764+0000","last_peered":"2024-09-21T02:05:54.799764+0000","last_clean":"2024-09-21T02:05:54.799764+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:05:54.799764+0000","last_undegraded":"2024-09-21T02:05:54.799764+0000","last_fullsized":"2024-09-21T02:05:54.799764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309190+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:54.309190+0000","last_peered":"2024-09-21T02:05:54.309190+0000","last_clean":"2024-09-21T02:05:54.309190+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:54.309190+0000","last_undegraded":"2024-09-21T02:05:54.309190+0000","last_fullsized":"2024-09-21T02:05:54.309190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751681+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:05:54.751681+0000","last_peered":"2024-09-21T02:05:54.751681+0000","last_clean":"2024-09-21T02:05:54.751681+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:05:54.751681+0000","last_undegraded":"2024-09-21T02:05:54.751681+0000","last_fullsized":"2024-09-21T02:05:54.751681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314237+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:05:54.314237+0000","last_peered":"2024-09-21T02:05:54.314237+0000","last_clean":"2024-09-21T02:05:54.314237+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:05:54.314237+0000","last_undegraded":"2024-09-21T02:05:54.314237+0000","last_fullsized":"2024-09-21T02:05:54.314237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840344+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:05:54.840344+0000","last_peered":"2024-09-21T02:05:54.840344+0000","last_clean":"2024-09-21T02:05:54.840344+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:05:54.840344+0000","last_undegraded":"2024-09-21T02:05:54.840344+0000","last_fullsized":"2024-09-21T02:05:54.840344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751927+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:05:54.751927+0000","last_peered":"2024-09-21T02:05:54.751927+0000","last_clean":"2024-09-21T02:05:54.751927+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:05:54.751927+0000","last_undegraded":"2024-09-21T02:05:54.751927+0000","last_fullsized":"2024-09-21T02:05:54.751927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752267+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:05:54.752267+0000","last_peered":"2024-09-21T02:05:54.752267+0000","last_clean":"2024-09-21T02:05:54.752267+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:05:54.752267+0000","last_undegraded":"2024-09-21T02:05:54.752267+0000","last_fullsized":"2024-09-21T02:05:54.752267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":74,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839541+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:54.839541+0000","last_peered":"2024-09-21T02:05:54.839541+0000","last_clean":"2024-09-21T02:05:54.839541+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:54.839541+0000","last_undegraded":"2024-09-21T02:05:54.839541+0000","last_fullsized":"2024-09-21T02:05:54.839541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309164+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:05:54.309164+0000","last_peered":"2024-09-21T02:05:54.309164+0000","last_clean":"2024-09-21T02:05:54.309164+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:05:54.309164+0000","last_undegraded":"2024-09-21T02:05:54.309164+0000","last_fullsized":"2024-09-21T02:05:54.309164+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799119+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:05:54.799119+0000","last_peered":"2024-09-21T02:05:54.799119+0000","last_clean":"2024-09-21T02:05:54.799119+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:05:54.799119+0000","last_undegraded":"2024-09-21T02:05:54.799119+0000","last_fullsized":"2024-09-21T02:05:54.799119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":101,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:11.821867+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:06:11.821867+0000","last_peered":"2024-09-21T02:06:11.821867+0000","last_clean":"2024-09-21T02:06:11.821867+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:06:11.821867+0000","last_undegraded":"2024-09-21T02:06:11.821867+0000","last_fullsized":"2024-09-21T02:06:11.821867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796362+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:05:54.796362+0000","last_peered":"2024-09-21T02:05:54.796362+0000","last_clean":"2024-09-21T02:05:54.796362+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:05:54.796362+0000","last_undegraded":"2024-09-21T02:05:54.796362+0000","last_fullsized":"2024-09-21T02:05:54.796362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799339+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:05:54.799339+0000","last_peered":"2024-09-21T02:05:54.799339+0000","last_clean":"2024-09-21T02:05:54.799339+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:05:54.799339+0000","last_undegraded":"2024-09-21T02:05:54.799339+0000","last_fullsized":"2024-09-21T02:05:54.799339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752282+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:54.752282+0000","last_peered":"2024-09-21T02:05:54.752282+0000","last_clean":"2024-09-21T02:05:54.752282+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:54.752282+0000","last_undegraded":"2024-09-21T02:05:54.752282+0000","last_fullsized":"2024-09-21T02:05:54.752282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314270+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:05:54.314270+0000","last_peered":"2024-09-21T02:05:54.314270+0000","last_clean":"2024-09-21T02:05:54.314270+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:05:54.314270+0000","last_undegraded":"2024-09-21T02:05:54.314270+0000","last_fullsized":"2024-09-21T02:05:54.314270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308650+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:05:54.308650+0000","last_peered":"2024-09-21T02:05:54.308650+0000","last_clean":"2024-09-21T02:05:54.308650+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:05:54.308650+0000","last_undegraded":"2024-09-21T02:05:54.308650+0000","last_fullsized":"2024-09-21T02:05:54.308650+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309809+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:54.309809+0000","last_peered":"2024-09-21T02:05:54.309809+0000","last_clean":"2024-09-21T02:05:54.309809+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:54.309809+0000","last_undegraded":"2024-09-21T02:05:54.309809+0000","last_fullsized":"2024-09-21T02:05:54.309809+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752687+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:05:54.752687+0000","last_peered":"2024-09-21T02:05:54.752687+0000","last_clean":"2024-09-21T02:05:54.752687+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:05:54.752687+0000","last_undegraded":"2024-09-21T02:05:54.752687+0000","last_fullsized":"2024-09-21T02:05:54.752687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840081+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:05:54.840081+0000","last_peered":"2024-09-21T02:05:54.840081+0000","last_clean":"2024-09-21T02:05:54.840081+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:05:54.840081+0000","last_undegraded":"2024-09-21T02:05:54.840081+0000","last_fullsized":"2024-09-21T02:05:54.840081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"84'34","reported_seq":188,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.835935+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:06:12.835935+0000","last_peered":"2024-09-21T02:06:12.835935+0000","last_clean":"2024-09-21T02:06:12.835935+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:06:12.835935+0000","last_undegraded":"2024-09-21T02:06:12.835935+0000","last_fullsized":"2024-09-21T02:06:12.835935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":122,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795664+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:05:54.795664+0000","last_peered":"2024-09-21T02:05:54.795664+0000","last_clean":"2024-09-21T02:05:54.795664+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:05:54.795664+0000","last_undegraded":"2024-09-21T02:05:54.795664+0000","last_fullsized":"2024-09-21T02:05:54.795664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839159+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:05:54.839159+0000","last_peered":"2024-09-21T02:05:54.839159+0000","last_clean":"2024-09-21T02:05:54.839159+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:05:54.839159+0000","last_undegraded":"2024-09-21T02:05:54.839159+0000","last_fullsized":"2024-09-21T02:05:54.839159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752902+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:54.752902+0000","last_peered":"2024-09-21T02:05:54.752902+0000","last_clean":"2024-09-21T02:05:54.752902+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:54.752902+0000","last_undegraded":"2024-09-21T02:05:54.752902+0000","last_fullsized":"2024-09-21T02:05:54.752902+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314110+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:05:54.314110+0000","last_peered":"2024-09-21T02:05:54.314110+0000","last_clean":"2024-09-21T02:05:54.314110+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:05:54.314110+0000","last_undegraded":"2024-09-21T02:05:54.314110+0000","last_fullsized":"2024-09-21T02:05:54.314110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838908+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:05:54.838908+0000","last_peered":"2024-09-21T02:05:54.838908+0000","last_clean":"2024-09-21T02:05:54.838908+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:05:54.838908+0000","last_undegraded":"2024-09-21T02:05:54.838908+0000","last_fullsized":"2024-09-21T02:05:54.838908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799598+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:54.799598+0000","last_peered":"2024-09-21T02:05:54.799598+0000","last_clean":"2024-09-21T02:05:54.799598+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:54.799598+0000","last_undegraded":"2024-09-21T02:05:54.799598+0000","last_fullsized":"2024-09-21T02:05:54.799598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839390+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:05:54.839390+0000","last_peered":"2024-09-21T02:05:54.839390+0000","last_clean":"2024-09-21T02:05:54.839390+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:05:54.839390+0000","last_undegraded":"2024-09-21T02:05:54.839390+0000","last_fullsized":"2024-09-21T02:05:54.839390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839423+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:05:54.839423+0000","last_peered":"2024-09-21T02:05:54.839423+0000","last_clean":"2024-09-21T02:05:54.839423+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:05:54.839423+0000","last_undegraded":"2024-09-21T02:05:54.839423+0000","last_fullsized":"2024-09-21T02:05:54.839423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":149,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.822897+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:06:12.822897+0000","last_peered":"2024-09-21T02:06:12.822897+0000","last_clean":"2024-09-21T02:06:12.822897+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:06:12.822897+0000","last_undegraded":"2024-09-21T02:06:12.822897+0000","last_fullsized":"2024-09-21T02:06:12.822897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693584+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:05:54.693584+0000","last_peered":"2024-09-21T02:05:54.693584+0000","last_clean":"2024-09-21T02:05:54.693584+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:05:54.693584+0000","last_undegraded":"2024-09-21T02:05:54.693584+0000","last_fullsized":"2024-09-21T02:05:54.693584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838626+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:05:54.838626+0000","last_peered":"2024-09-21T02:05:54.838626+0000","last_clean":"2024-09-21T02:05:54.838626+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:05:54.838626+0000","last_undegraded":"2024-09-21T02:05:54.838626+0000","last_fullsized":"2024-09-21T02:05:54.838626+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795480+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:05:54.795480+0000","last_peered":"2024-09-21T02:05:54.795480+0000","last_clean":"2024-09-21T02:05:54.795480+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:05:54.795480+0000","last_undegraded":"2024-09-21T02:05:54.795480+0000","last_fullsized":"2024-09-21T02:05:54.795480+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752188+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:54.752188+0000","last_peered":"2024-09-21T02:05:54.752188+0000","last_clean":"2024-09-21T02:05:54.752188+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:54.752188+0000","last_undegraded":"2024-09-21T02:05:54.752188+0000","last_fullsized":"2024-09-21T02:05:54.752188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841549+0000","last_change":"2024-09-21T02:05:54.841426+0000","last_active":"2024-09-21T02:05:54.841549+0000","last_peered":"2024-09-21T02:05:54.841549+0000","last_clean":"2024-09-21T02:05:54.841549+0000","last_became_active":"2024-09-21T02:05:54.840855+0000","last_became_peered":"2024-09-21T02:05:54.840855+0000","last_unstale":"2024-09-21T02:05:54.841549+0000","last_undegraded":"2024-09-21T02:05:54.841549+0000","last_fullsized":"2024-09-21T02:05:54.841549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:40.637427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693707+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:05:54.693707+0000","last_peered":"2024-09-21T02:05:54.693707+0000","last_clean":"2024-09-21T02:05:54.693707+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:05:54.693707+0000","last_undegraded":"2024-09-21T02:05:54.693707+0000","last_fullsized":"2024-09-21T02:05:54.693707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308544+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:54.308544+0000","last_peered":"2024-09-21T02:05:54.308544+0000","last_clean":"2024-09-21T02:05:54.308544+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:54.308544+0000","last_undegraded":"2024-09-21T02:05:54.308544+0000","last_fullsized":"2024-09-21T02:05:54.308544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309450+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:54.309450+0000","last_peered":"2024-09-21T02:05:54.309450+0000","last_clean":"2024-09-21T02:05:54.309450+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:54.309450+0000","last_undegraded":"2024-09-21T02:05:54.309450+0000","last_fullsized":"2024-09-21T02:05:54.309450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752299+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:05:54.752299+0000","last_peered":"2024-09-21T02:05:54.752299+0000","last_clean":"2024-09-21T02:05:54.752299+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:05:54.752299+0000","last_undegraded":"2024-09-21T02:05:54.752299+0000","last_fullsized":"2024-09-21T02:05:54.752299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309752+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:05:54.309752+0000","last_peered":"2024-09-21T02:05:54.309752+0000","last_clean":"2024-09-21T02:05:54.309752+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:05:54.309752+0000","last_undegraded":"2024-09-21T02:05:54.309752+0000","last_fullsized":"2024-09-21T02:05:54.309752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'6","reported_seq":102,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:12.823811+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:06:12.823811+0000","last_peered":"2024-09-21T02:06:12.823811+0000","last_clean":"2024-09-21T02:06:12.823811+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:06:12.823811+0000","last_undegraded":"2024-09-21T02:06:12.823811+0000","last_fullsized":"2024-09-21T02:06:12.823811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839609+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:05:54.839609+0000","last_peered":"2024-09-21T02:05:54.839609+0000","last_clean":"2024-09-21T02:05:54.839609+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:05:54.839609+0000","last_undegraded":"2024-09-21T02:05:54.839609+0000","last_fullsized":"2024-09-21T02:05:54.839609+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309832+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:05:54.309832+0000","last_peered":"2024-09-21T02:05:54.309832+0000","last_clean":"2024-09-21T02:05:54.309832+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:05:54.309832+0000","last_undegraded":"2024-09-21T02:05:54.309832+0000","last_fullsized":"2024-09-21T02:05:54.309832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313822+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:05:54.313822+0000","last_peered":"2024-09-21T02:05:54.313822+0000","last_clean":"2024-09-21T02:05:54.313822+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:05:54.313822+0000","last_undegraded":"2024-09-21T02:05:54.313822+0000","last_fullsized":"2024-09-21T02:05:54.313822+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309878+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:05:54.309878+0000","last_peered":"2024-09-21T02:05:54.309878+0000","last_clean":"2024-09-21T02:05:54.309878+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:05:54.309878+0000","last_undegraded":"2024-09-21T02:05:54.309878+0000","last_fullsized":"2024-09-21T02:05:54.309878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309595+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:05:54.309595+0000","last_peered":"2024-09-21T02:05:54.309595+0000","last_clean":"2024-09-21T02:05:54.309595+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:05:54.309595+0000","last_undegraded":"2024-09-21T02:05:54.309595+0000","last_fullsized":"2024-09-21T02:05:54.309595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838375+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:05:54.838375+0000","last_peered":"2024-09-21T02:05:54.838375+0000","last_clean":"2024-09-21T02:05:54.838375+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:05:54.838375+0000","last_undegraded":"2024-09-21T02:05:54.838375+0000","last_fullsized":"2024-09-21T02:05:54.838375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308869+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:05:54.308869+0000","last_peered":"2024-09-21T02:05:54.308869+0000","last_clean":"2024-09-21T02:05:54.308869+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:05:54.308869+0000","last_undegraded":"2024-09-21T02:05:54.308869+0000","last_fullsized":"2024-09-21T02:05:54.308869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314470+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:05:54.314470+0000","last_peered":"2024-09-21T02:05:54.314470+0000","last_clean":"2024-09-21T02:05:54.314470+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:05:54.314470+0000","last_undegraded":"2024-09-21T02:05:54.314470+0000","last_fullsized":"2024-09-21T02:05:54.314470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310118+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:05:54.310118+0000","last_peered":"2024-09-21T02:05:54.310118+0000","last_clean":"2024-09-21T02:05:54.310118+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:05:54.310118+0000","last_undegraded":"2024-09-21T02:05:54.310118+0000","last_fullsized":"2024-09-21T02:05:54.310118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694096+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:05:54.694096+0000","last_peered":"2024-09-21T02:05:54.694096+0000","last_clean":"2024-09-21T02:05:54.694096+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:05:54.694096+0000","last_undegraded":"2024-09-21T02:05:54.694096+0000","last_fullsized":"2024-09-21T02:05:54.694096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751904+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:05:54.751904+0000","last_peered":"2024-09-21T02:05:54.751904+0000","last_clean":"2024-09-21T02:05:54.751904+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:05:54.751904+0000","last_undegraded":"2024-09-21T02:05:54.751904+0000","last_fullsized":"2024-09-21T02:05:54.751904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693654+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:05:54.693654+0000","last_peered":"2024-09-21T02:05:54.693654+0000","last_clean":"2024-09-21T02:05:54.693654+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:05:54.693654+0000","last_undegraded":"2024-09-21T02:05:54.693654+0000","last_fullsized":"2024-09-21T02:05:54.693654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839095+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:05:54.839095+0000","last_peered":"2024-09-21T02:05:54.839095+0000","last_clean":"2024-09-21T02:05:54.839095+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:05:54.839095+0000","last_undegraded":"2024-09-21T02:05:54.839095+0000","last_fullsized":"2024-09-21T02:05:54.839095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308899+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:05:54.308899+0000","last_peered":"2024-09-21T02:05:54.308899+0000","last_clean":"2024-09-21T02:05:54.308899+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:05:54.308899+0000","last_undegraded":"2024-09-21T02:05:54.308899+0000","last_fullsized":"2024-09-21T02:05:54.308899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314056+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:05:54.314056+0000","last_peered":"2024-09-21T02:05:54.314056+0000","last_clean":"2024-09-21T02:05:54.314056+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:05:54.314056+0000","last_undegraded":"2024-09-21T02:05:54.314056+0000","last_fullsized":"2024-09-21T02:05:54.314056+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310008+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:05:54.310008+0000","last_peered":"2024-09-21T02:05:54.310008+0000","last_clean":"2024-09-21T02:05:54.310008+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:05:54.310008+0000","last_undegraded":"2024-09-21T02:05:54.310008+0000","last_fullsized":"2024-09-21T02:05:54.310008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796157+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:05:54.796157+0000","last_peered":"2024-09-21T02:05:54.796157+0000","last_clean":"2024-09-21T02:05:54.796157+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:05:54.796157+0000","last_undegraded":"2024-09-21T02:05:54.796157+0000","last_fullsized":"2024-09-21T02:05:54.796157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314535+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:05:54.314535+0000","last_peered":"2024-09-21T02:05:54.314535+0000","last_clean":"2024-09-21T02:05:54.314535+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:05:54.314535+0000","last_undegraded":"2024-09-21T02:05:54.314535+0000","last_fullsized":"2024-09-21T02:05:54.314535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838862+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:05:54.838862+0000","last_peered":"2024-09-21T02:05:54.838862+0000","last_clean":"2024-09-21T02:05:54.838862+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:05:54.838862+0000","last_undegraded":"2024-09-21T02:05:54.838862+0000","last_fullsized":"2024-09-21T02:05:54.838862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838248+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:05:54.838248+0000","last_peered":"2024-09-21T02:05:54.838248+0000","last_clean":"2024-09-21T02:05:54.838248+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:05:54.838248+0000","last_undegraded":"2024-09-21T02:05:54.838248+0000","last_fullsized":"2024-09-21T02:05:54.838248+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838241+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:05:54.838241+0000","last_peered":"2024-09-21T02:05:54.838241+0000","last_clean":"2024-09-21T02:05:54.838241+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:05:54.838241+0000","last_undegraded":"2024-09-21T02:05:54.838241+0000","last_fullsized":"2024-09-21T02:05:54.838241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308871+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:05:54.308871+0000","last_peered":"2024-09-21T02:05:54.308871+0000","last_clean":"2024-09-21T02:05:54.308871+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:05:54.308871+0000","last_undegraded":"2024-09-21T02:05:54.308871+0000","last_fullsized":"2024-09-21T02:05:54.308871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314437+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:05:54.314437+0000","last_peered":"2024-09-21T02:05:54.314437+0000","last_clean":"2024-09-21T02:05:54.314437+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:05:54.314437+0000","last_undegraded":"2024-09-21T02:05:54.314437+0000","last_fullsized":"2024-09-21T02:05:54.314437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799721+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:05:54.799721+0000","last_peered":"2024-09-21T02:05:54.799721+0000","last_clean":"2024-09-21T02:05:54.799721+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:05:54.799721+0000","last_undegraded":"2024-09-21T02:05:54.799721+0000","last_fullsized":"2024-09-21T02:05:54.799721+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.310103+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:05:54.310103+0000","last_peered":"2024-09-21T02:05:54.310103+0000","last_clean":"2024-09-21T02:05:54.310103+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:05:54.310103+0000","last_undegraded":"2024-09-21T02:05:54.310103+0000","last_fullsized":"2024-09-21T02:05:54.310103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839470+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:05:54.839470+0000","last_peered":"2024-09-21T02:05:54.839470+0000","last_clean":"2024-09-21T02:05:54.839470+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:05:54.839470+0000","last_undegraded":"2024-09-21T02:05:54.839470+0000","last_fullsized":"2024-09-21T02:05:54.839470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693644+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:05:54.693644+0000","last_peered":"2024-09-21T02:05:54.693644+0000","last_clean":"2024-09-21T02:05:54.693644+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:05:54.693644+0000","last_undegraded":"2024-09-21T02:05:54.693644+0000","last_fullsized":"2024-09-21T02:05:54.693644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799161+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:05:54.799161+0000","last_peered":"2024-09-21T02:05:54.799161+0000","last_clean":"2024-09-21T02:05:54.799161+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:05:54.799161+0000","last_undegraded":"2024-09-21T02:05:54.799161+0000","last_fullsized":"2024-09-21T02:05:54.799161+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309453+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:05:54.309453+0000","last_peered":"2024-09-21T02:05:54.309453+0000","last_clean":"2024-09-21T02:05:54.309453+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:05:54.309453+0000","last_undegraded":"2024-09-21T02:05:54.309453+0000","last_fullsized":"2024-09-21T02:05:54.309453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839817+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:05:54.839817+0000","last_peered":"2024-09-21T02:05:54.839817+0000","last_clean":"2024-09-21T02:05:54.839817+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:05:54.839817+0000","last_undegraded":"2024-09-21T02:05:54.839817+0000","last_fullsized":"2024-09-21T02:05:54.839817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314480+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:05:54.314480+0000","last_peered":"2024-09-21T02:05:54.314480+0000","last_clean":"2024-09-21T02:05:54.314480+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:05:54.314480+0000","last_undegraded":"2024-09-21T02:05:54.314480+0000","last_fullsized":"2024-09-21T02:05:54.314480+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838525+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:05:54.838525+0000","last_peered":"2024-09-21T02:05:54.838525+0000","last_clean":"2024-09-21T02:05:54.838525+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:05:54.838525+0000","last_undegraded":"2024-09-21T02:05:54.838525+0000","last_fullsized":"2024-09-21T02:05:54.838525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838614+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:05:54.838614+0000","last_peered":"2024-09-21T02:05:54.838614+0000","last_clean":"2024-09-21T02:05:54.838614+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:05:54.838614+0000","last_undegraded":"2024-09-21T02:05:54.838614+0000","last_fullsized":"2024-09-21T02:05:54.838614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314403+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:05:54.314403+0000","last_peered":"2024-09-21T02:05:54.314403+0000","last_clean":"2024-09-21T02:05:54.314403+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:05:54.314403+0000","last_undegraded":"2024-09-21T02:05:54.314403+0000","last_fullsized":"2024-09-21T02:05:54.314403+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74198,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":586,"num_read_kb":374,"num_write":251,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":222594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710754,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":500919,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874278,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1105920,"data_stored":1153835,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":504442,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":500612,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397558,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1024000,"data_stored":1098003,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561084,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":950272,"data_stored":496241,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1032192,"data_stored":1092803,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":958464,"data_stored":500315,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117068,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":962560,"data_stored":495558,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378705,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1036288,"data_stored":563982,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542230,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1032192,"data_stored":563595,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":974848,"data_stored":497697,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:06:19.979 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:06:19.979 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T02:06:19.979 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T02:06:19.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-21T02:06:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:19 smithi016 ceph-mon[31245]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:20.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:19 smithi161 ceph-mon[34061]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:20.242 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:19 smithi123 ceph-mon[33453]: pgmap v327: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 -- 172.21.15.16:0/3337501931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba98104a30 msgr2=0x7fba98104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 --2- 172.21.15.16:0/3337501931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba98104a30 0x7fba98104e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fba94009900 tx=0x7fba9402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 -- 172.21.15.16:0/3337501931 shutdown_connections 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 --2- 172.21.15.16:0/3337501931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba9810de60 0x7fba98110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 --2- 172.21.15.16:0/3337501931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98105380 0x7fba9810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 --2- 172.21.15.16:0/3337501931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba98104a30 0x7fba98104e40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 -- 172.21.15.16:0/3337501931 >> 172.21.15.16:0/3337501931 conn(0x7fba98079760 msgr2=0x7fba98079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 -- 172.21.15.16:0/3337501931 shutdown_connections 2024-09-21T02:06:20.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 -- 172.21.15.16:0/3337501931 wait complete. 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.712+0000 7fbaa069d640 1 Processor -- start 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 -- start start 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 0x7fba9819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba98105380 0x7fba9819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba98113950 con 0x7fba9810de60 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba981137d0 con 0x7fba98105380 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fbaa069d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fba98113ad0 con 0x7fba98104a30 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38860/0 (socket says 172.21.15.16:38860) 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 -- 172.21.15.16:0/2229836439 learned_addr learned my addr 172.21.15.16:0/2229836439 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9dc38640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba98105380 0x7fba9819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9e439640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 0x7fba9819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 -- 172.21.15.16:0/2229836439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 msgr2=0x7fba9819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 0x7fba9819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 -- 172.21.15.16:0/2229836439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba98105380 msgr2=0x7fba9819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba98105380 0x7fba9819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9ec3a640 1 -- 172.21.15.16:0/2229836439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba981a02f0 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.713+0000 7fba9e439640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 0x7fba9819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fba9ec3a640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fba9000eaa0 tx=0x7fba9000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90004d20 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba981a05e0 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba90005020 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900116d0 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.714+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba981a6e00 con 0x7fba9810de60 2024-09-21T02:06:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.715+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba9806c1f0 con 0x7fba9810de60 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.717+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fba900051d0 con 0x7fba9810de60 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.717+0000 7fba8f7fe640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 0x7fba840b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.717+0000 7fba9e439640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 0x7fba840b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.717+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fba900e7f10 con 0x7fba9810de60 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.718+0000 7fba9e439640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 0x7fba840b8b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba94006010 tx=0x7fba94000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.721+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fba90016030 con 0x7fba9810de60 2024-09-21T02:06:20.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:20 smithi016 ceph-mon[31245]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.955+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fba98104e40 con 0x7fba9810de60 2024-09-21T02:06:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.956+0000 7fba8f7fe640 1 -- 172.21.15.16:0/2229836439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fba98104e40 con 0x7fba9810de60 2024-09-21T02:06:20.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:20.957 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T02:06:20.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 msgr2=0x7fba840b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 0x7fba840b8b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba94006010 tx=0x7fba94000bc0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 msgr2=0x7fba9819fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fba9000eaa0 tx=0x7fba9000eea0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 shutdown_connections 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fba840b6620 0x7fba840b8b30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba9810de60 0x7fba9819fc80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba98105380 0x7fba9819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 --2- 172.21.15.16:0/2229836439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba98104a30 0x7fba9819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 >> 172.21.15.16:0/2229836439 conn(0x7fba98079760 msgr2=0x7fba98100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 shutdown_connections 2024-09-21T02:06:20.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:20.958+0000 7fbaa069d640 1 -- 172.21.15.16:0/2229836439 wait complete. 2024-09-21T02:06:21.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:20 smithi161 ceph-mon[34061]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:20 smithi123 ceph-mon[33453]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:06:21.787 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T02:06:21.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:06:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:21 smithi016 ceph-mon[31245]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:21 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2229836439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:06:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:21 smithi161 ceph-mon[34061]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:21 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2229836439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:06:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:21 smithi123 ceph-mon[33453]: pgmap v328: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:21 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2229836439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T02:06:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.521+0000 7f5d79278640 1 -- 172.21.15.16:0/1999507337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 msgr2=0x7f5d7410ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.521+0000 7f5d79278640 1 --2- 172.21.15.16:0/1999507337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d7410ed60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009900 tx=0x7f5d6c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 -- 172.21.15.16:0/1999507337 shutdown_connections 2024-09-21T02:06:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 --2- 172.21.15.16:0/1999507337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d7410ed60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 --2- 172.21.15.16:0/1999507337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d74107be0 0x7f5d74107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 --2- 172.21.15.16:0/1999507337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d741022a0 0x7f5d741026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 -- 172.21.15.16:0/1999507337 >> 172.21.15.16:0/1999507337 conn(0x7f5d74079ec0 msgr2=0x7f5d7407a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 -- 172.21.15.16:0/1999507337 shutdown_connections 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 -- 172.21.15.16:0/1999507337 wait complete. 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.522+0000 7f5d79278640 1 Processor -- start 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 -- start start 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d741022a0 0x7f5d74110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d74107be0 0x7f5d74110b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113a30 con 0x7f5d74108530 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741138b0 con 0x7f5d741022a0 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d79278640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74113bb0 con 0x7f5d74107be0 2024-09-21T02:06:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38876/0 (socket says 172.21.15.16:38876) 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 -- 172.21.15.16:0/2874196243 learned_addr learned my addr 172.21.15.16:0/2874196243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d72ffd640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d741022a0 0x7f5d74110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 -- 172.21.15.16:0/2874196243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d74107be0 msgr2=0x7f5d74110b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d74107be0 0x7f5d74110b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 -- 172.21.15.16:0/2874196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d741022a0 msgr2=0x7f5d74110650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d741022a0 0x7f5d74110650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.523+0000 7f5d737fe640 1 -- 172.21.15.16:0/2874196243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d741a86a0 con 0x7f5d74108530 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d737fe640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005ab0 tx=0x7f5d6c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c03f030 con 0x7f5d74108530 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d741a8930 con 0x7f5d74108530 2024-09-21T02:06:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d6c003a80 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c035070 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.524+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d741a8e60 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.525+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d741026b0 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.527+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f5d6c042030 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.528+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 0x7f5d5c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.528+0000 7f5d72ffd640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 0x7f5d5c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.528+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f5d6c1082f0 con 0x7f5d74108530 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.528+0000 7f5d72ffd640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 0x7f5d5c0b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d741120a0 tx=0x7f5d64009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.532+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5d6c0ce590 con 0x7f5d74108530 2024-09-21T02:06:22.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.755+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d74111950 con 0x7f5d74108530 2024-09-21T02:06:22.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.756+0000 7f5d5bfff640 1 -- 172.21.15.16:0/2874196243 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7f5d6c0d2970 con 0x7f5d74108530 2024-09-21T02:06:22.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:22.757 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:06:22.757 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:22.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 msgr2=0x7f5d5c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:22.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 0x7f5d5c0b8b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d741120a0 tx=0x7f5d64009680 comp rx=0 tx=0).stop 2024-09-21T02:06:22.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 msgr2=0x7f5d741110d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005ab0 tx=0x7f5d6c0040e0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 shutdown_connections 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d5c0b65f0 0x7f5d5c0b8b00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d74108530 0x7f5d741110d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d74107be0 0x7f5d74110b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 --2- 172.21.15.16:0/2874196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d741022a0 0x7f5d74110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.758+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 >> 172.21.15.16:0/2874196243 conn(0x7f5d74079ec0 msgr2=0x7f5d74105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.759+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 shutdown_connections 2024-09-21T02:06:22.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:22.759+0000 7f5d79278640 1 -- 172.21.15.16:0/2874196243 wait complete. 2024-09-21T02:06:23.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:22 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2874196243' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:23.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:22 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2874196243' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:23.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:22 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2874196243' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:23.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:06:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:24.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:23 smithi016 ceph-mon[31245]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:24.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:23 smithi161 ceph-mon[34061]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:23 smithi123 ceph-mon[33453]: pgmap v329: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 -- 172.21.15.16:0/2778202949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec1083d0 msgr2=0x7fe5ec1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 --2- 172.21.15.16:0/2778202949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec1087e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009900 tx=0x7fe5dc02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 -- 172.21.15.16:0/2778202949 shutdown_connections 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 --2- 172.21.15.16:0/2778202949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 --2- 172.21.15.16:0/2778202949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec1087e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.273+0000 7fe5f314a640 1 --2- 172.21.15.16:0/2778202949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.274+0000 7fe5f314a640 1 -- 172.21.15.16:0/2778202949 >> 172.21.15.16:0/2778202949 conn(0x7fe5ec079760 msgr2=0x7fe5ec079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.274+0000 7fe5f314a640 1 -- 172.21.15.16:0/2778202949 shutdown_connections 2024-09-21T02:06:24.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.274+0000 7fe5f314a640 1 -- 172.21.15.16:0/2778202949 wait complete. 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.274+0000 7fe5f314a640 1 Processor -- start 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.274+0000 7fe5f314a640 1 -- start start 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec112840 con 0x7fe5ec102ae0 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec1126c0 con 0x7fe5ec1083d0 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f314a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec1129c0 con 0x7fe5ec108d20 2024-09-21T02:06:24.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38898/0 (socket says 172.21.15.16:38898) 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 -- 172.21.15.16:0/242878709 learned_addr learned my addr 172.21.15.16:0/242878709 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5ebfff640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f16e7640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 -- 172.21.15.16:0/242878709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 msgr2=0x7fe5ec19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 -- 172.21.15.16:0/242878709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec1083d0 msgr2=0x7fe5ec19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f0ee6640 1 -- 172.21.15.16:0/242878709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec1a0270 con 0x7fe5ec102ae0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.275+0000 7fe5f16e7640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5f0ee6640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c920 tx=0x7fe5d400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4007c70 con 0x7fe5ec102ae0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d4007e10 con 0x7fe5ec102ae0 2024-09-21T02:06:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec1a0560 con 0x7fe5ec102ae0 2024-09-21T02:06:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4018710 con 0x7fe5ec102ae0 2024-09-21T02:06:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.276+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1a6d80 con 0x7fe5ec102ae0 2024-09-21T02:06:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.277+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec103d10 con 0x7fe5ec102ae0 2024-09-21T02:06:24.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.279+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fe5d400f450 con 0x7fe5ec102ae0 2024-09-21T02:06:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.279+0000 7fe5e9ffb640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 0x7fe5d80b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.279+0000 7fe5ebfff640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 0x7fe5d80b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.279+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fe5d40e7a80 con 0x7fe5ec102ae0 2024-09-21T02:06:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.280+0000 7fe5ebfff640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 0x7fe5d80b8bb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc004770 tx=0x7fe5dc0058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:24.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.284+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe5d4014030 con 0x7fe5ec102ae0 2024-09-21T02:06:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.512+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5ec10bb20 con 0x7fe5ec102ae0 2024-09-21T02:06:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.513+0000 7fe5e9ffb640 1 -- 172.21.15.16:0/242878709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7fe5d40add20 con 0x7fe5ec102ae0 2024-09-21T02:06:24.514 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:24.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:06:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 msgr2=0x7fe5d80b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 0x7fe5d80b8bb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc004770 tx=0x7fe5dc0058d0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 msgr2=0x7fe5ec19a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c920 tx=0x7fe5d400cde0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 shutdown_connections 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe5d80b66a0 0x7fe5d80b8bb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5ec108d20 0x7fe5ec19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5ec1083d0 0x7fe5ec19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 --2- 172.21.15.16:0/242878709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5ec102ae0 0x7fe5ec19a3c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 >> 172.21.15.16:0/242878709 conn(0x7fe5ec079760 msgr2=0x7fe5ec06bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 shutdown_connections 2024-09-21T02:06:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:24.515+0000 7fe5f314a640 1 -- 172.21.15.16:0/242878709 wait complete. 2024-09-21T02:06:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:24 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/242878709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:25.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:24 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/242878709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:24 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/242878709' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:25.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:06:25.594 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:26.009 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:25 smithi016 ceph-mon[31245]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 -- 172.21.15.16:0/3297705250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 msgr2=0x7f450c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 --2- 172.21.15.16:0/3297705250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c110330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4504009ca0 tx=0x7f450402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 -- 172.21.15.16:0/3297705250 shutdown_connections 2024-09-21T02:06:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 --2- 172.21.15.16:0/3297705250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c110330 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 --2- 172.21.15.16:0/3297705250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 0x7f450c10d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 --2- 172.21.15.16:0/3297705250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 0x7f450c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 -- 172.21.15.16:0/3297705250 >> 172.21.15.16:0/3297705250 conn(0x7f450c079760 msgr2=0x7f450c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 -- 172.21.15.16:0/3297705250 shutdown_connections 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.101+0000 7f4511b4b640 1 -- 172.21.15.16:0/3297705250 wait complete. 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 Processor -- start 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 -- start start 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 0x7f450c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 0x7f450c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f450c1139a0 con 0x7f450c10de60 2024-09-21T02:06:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f450c113820 con 0x7f450c105380 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f4511b4b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f450c113b20 con 0x7f450c104a30 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f450bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f450bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38906/0 (socket says 172.21.15.16:38906) 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f450bfff640 1 -- 172.21.15.16:0/1280808140 learned_addr learned my addr 172.21.15.16:0/1280808140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.102+0000 7f450b7fe640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 0x7f450c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450affd640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 0x7f450c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 -- 172.21.15.16:0/1280808140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 msgr2=0x7f450c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 0x7f450c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 -- 172.21.15.16:0/1280808140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 msgr2=0x7f450c19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 0x7f450c19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 -- 172.21.15.16:0/1280808140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c1a0180 con 0x7f450c10de60 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f450bfff640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4504005cc0 tx=0x7f4504032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504004040 con 0x7f450c10de60 2024-09-21T02:06:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4504004670 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450c1a0410 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504030cc0 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.103+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450c1a6c90 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.105+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450c06c1f0 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.106+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f45040041e0 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.107+0000 7f4508ff9640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 0x7f44f40b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.107+0000 7f450b7fe640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 0x7f44f40b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.107+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f4504108c30 con 0x7f450c10de60 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.107+0000 7f450b7fe640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 0x7f44f40b8bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f450c06c8f0 tx=0x7f44fc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.111+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f45040ceed0 con 0x7f450c10de60 2024-09-21T02:06:26.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:25 smithi161 ceph-mon[34061]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:25 smithi123 ceph-mon[33453]: pgmap v330: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.299+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f450c1a6e60 con 0x7f450c10de60 2024-09-21T02:06:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.301+0000 7f4508ff9640 1 -- 172.21.15.16:0/1280808140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+25198 (secure 0 0 0) 0x7f45040d32b0 con 0x7f450c10de60 2024-09-21T02:06:26.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:26.302 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:05:54.299820+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:06:26.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 msgr2=0x7f44f40b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:26.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 0x7f44f40b8bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f450c06c8f0 tx=0x7f44fc005f90 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 msgr2=0x7f450c19fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4504005cc0 tx=0x7f4504032c40 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 shutdown_connections 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44f40b66a0 0x7f44f40b8bb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f450c10de60 0x7f450c19fb10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c105380 0x7f450c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 --2- 172.21.15.16:0/1280808140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f450c104a30 0x7f450c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 >> 172.21.15.16:0/1280808140 conn(0x7f450c079760 msgr2=0x7f450c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 shutdown_connections 2024-09-21T02:06:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:26.304+0000 7f4511b4b640 1 -- 172.21.15.16:0/1280808140 wait complete. 2024-09-21T02:06:26.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1280808140' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:27.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:06:27.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1280808140' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1280808140' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 -- 172.21.15.16:0/301642138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108420 msgr2=0x7f2e3c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/301642138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108420 0x7f2e3c108830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009900 tx=0x7f2e2c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 -- 172.21.15.16:0/301642138 shutdown_connections 2024-09-21T02:06:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/301642138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108d70 0x7f2e3c10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/301642138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108420 0x7f2e3c108830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/301642138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 0x7f2e3c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 -- 172.21.15.16:0/301642138 >> 172.21.15.16:0/301642138 conn(0x7f2e3c079770 msgr2=0x7f2e3c079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 -- 172.21.15.16:0/301642138 shutdown_connections 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.886+0000 7f2e43e2d640 1 -- 172.21.15.16:0/301642138 wait complete. 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 Processor -- start 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 -- start start 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 0x7f2e3c19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108420 0x7f2e3c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1128e0 con 0x7f2e3c108d70 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c112760 con 0x7f2e3c102ae0 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e43e2d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c112a60 con 0x7f2e3c108420 2024-09-21T02:06:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e423ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e423ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49438/0 (socket says 172.21.15.16:49438) 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e423ca640 1 -- 172.21.15.16:0/199849271 learned_addr learned my addr 172.21.15.16:0/199849271 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.887+0000 7f2e413c8640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108420 0x7f2e3c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e41bc9640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 0x7f2e3c19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 -- 172.21.15.16:0/199849271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108420 msgr2=0x7f2e3c19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108420 0x7f2e3c19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 -- 172.21.15.16:0/199849271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 msgr2=0x7f2e3c19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 0x7f2e3c19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 -- 172.21.15.16:0/199849271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3c1a0450 con 0x7f2e3c108d70 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e423ca640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400eaa0 tx=0x7f2e3400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400cdb0 con 0x7f2e3c108d70 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3c1a0740 con 0x7f2e3c108d70 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e34005020 con 0x7f2e3c108d70 2024-09-21T02:06:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e340116d0 con 0x7f2e3c108d70 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.888+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3c1a6ea0 con 0x7f2e3c108d70 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.889+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e3c103d60 con 0x7f2e3c108d70 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.891+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2e34004a40 con 0x7f2e3c108d70 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.892+0000 7f2e32ffd640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 0x7f2e280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.892+0000 7f2e41bc9640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 0x7f2e280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.892+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2e340e7e70 con 0x7f2e3c108d70 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.892+0000 7f2e41bc9640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 0x7f2e280b8b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2e38005fd0 tx=0x7f2e3800b590 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:27.895+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2e34016030 con 0x7f2e3c108d70 2024-09-21T02:06:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.120+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e3c102ef0 con 0x7f2e3c108d70 2024-09-21T02:06:28.121 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:27 smithi016 ceph-mon[31245]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.121+0000 7f2e32ffd640 1 -- 172.21.15.16:0/199849271 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7f2e3c102ef0 con 0x7f2e3c108d70 2024-09-21T02:06:28.122 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:28.122 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:06:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 msgr2=0x7f2e280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 0x7f2e280b8b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2e38005fd0 tx=0x7f2e3800b590 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 msgr2=0x7f2e3c19fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400eaa0 tx=0x7f2e3400ef60 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 shutdown_connections 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2e280b65f0 0x7f2e280b8b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3c108d70 0x7f2e3c19fd20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e3c108420 0x7f2e3c19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 --2- 172.21.15.16:0/199849271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3c102ae0 0x7f2e3c19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 >> 172.21.15.16:0/199849271 conn(0x7f2e3c079770 msgr2=0x7f2e3c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 shutdown_connections 2024-09-21T02:06:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:28.124+0000 7f2e43e2d640 1 -- 172.21.15.16:0/199849271 wait complete. 2024-09-21T02:06:28.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:27 smithi161 ceph-mon[34061]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:27 smithi123 ceph-mon[33453]: pgmap v331: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:28.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:04:34.359195+0000', 'modified': '2024-09-21T02:05:20.732006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24710, 'mds_1': 24683, 'mds_2': 24667, 'mds_3': 24659, 'mds_4': 24655}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14841': {'gid': 14841, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/1623043260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1623043260}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1623043260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14853': {'gid': 14853, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/3323661058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 3323661058}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 3323661058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24595': {'gid': 24595, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6833/2556025136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6832', 'nonce': 2556025136}, {'type': 'v1', 'addr': '172.21.15.161:6833', 'nonce': 2556025136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24626': {'gid': 24626, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6833/3641506877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6832', 'nonce': 3641506877}, {'type': 'v1', 'addr': '172.21.15.123:6833', 'nonce': 3641506877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24643': {'gid': 24643, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/1279942273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 1279942273}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 1279942273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24655': {'gid': 24655, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.161:6837/2373808610', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2373808610}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2373808610}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24659': {'gid': 24659, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.123:6835/2421556237', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2421556237}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2421556237}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24667': {'gid': 24667, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2696881795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2696881795}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2696881795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24683': {'gid': 24683, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/1023560595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 1023560595}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 1023560595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24710': {'gid': 24710, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/1116723582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1116723582}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1116723582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24710, 'qdb_cluster': [24655, 24659, 24667, 24683, 24710]} 2024-09-21T02:06:28.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:06:28.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:06:29.103 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/199849271' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:29.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/199849271' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/199849271' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 -- 172.21.15.16:0/3876326320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 msgr2=0x7f25741098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 --2- 172.21.15.16:0/3876326320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741098d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2560009900 tx=0x7f256002a410 comp rx=0 tx=0).stop 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 -- 172.21.15.16:0/3876326320 shutdown_connections 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 --2- 172.21.15.16:0/3876326320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574109e10 0x7f2574110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 --2- 172.21.15.16:0/3876326320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741098d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 --2- 172.21.15.16:0/3876326320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574101fe0 0x7f25741023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 -- 172.21.15.16:0/3876326320 >> 172.21.15.16:0/3876326320 conn(0x7f2574077bc0 msgr2=0x7f2574078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 -- 172.21.15.16:0/3876326320 shutdown_connections 2024-09-21T02:06:29.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 -- 172.21.15.16:0/3876326320 wait complete. 2024-09-21T02:06:29.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.615+0000 7f257aa65640 1 Processor -- start 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 -- start start 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574101fe0 0x7f2574102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574109e10 0x7f2574103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113980 con 0x7f25741094c0 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113800 con 0x7f2574101fe0 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f257aa65640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2574113b00 con 0x7f2574109e10 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49458/0 (socket says 172.21.15.16:49458) 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 -- 172.21.15.16:0/956654208 learned_addr learned my addr 172.21.15.16:0/956654208 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2578801640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574101fe0 0x7f2574102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2579002640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574109e10 0x7f2574103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 -- 172.21.15.16:0/956654208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574109e10 msgr2=0x7f2574103a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574109e10 0x7f2574103a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 -- 172.21.15.16:0/956654208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574101fe0 msgr2=0x7f2574102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574101fe0 0x7f2574102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.616+0000 7f2573fff640 1 -- 172.21.15.16:0/956654208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25741a8600 con 0x7f25741094c0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f2573fff640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f256002a930 tx=0x7f2560003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f2571ffb640 1 -- 172.21.15.16:0/956654208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256003f070 con 0x7f25741094c0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f2571ffb640 1 -- 172.21.15.16:0/956654208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f256003bd00 con 0x7f25741094c0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25741a8890 con 0x7f25741094c0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f2571ffb640 1 -- 172.21.15.16:0/956654208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560035030 con 0x7f25741094c0 2024-09-21T02:06:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.617+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25741a8d70 con 0x7f25741094c0 2024-09-21T02:06:29.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 Processor -- start 2024-09-21T02:06:29.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 -- start start 2024-09-21T02:06:29.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2571ffb640 1 -- 172.21.15.16:0/956654208 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2560042070 con 0x7f25741094c0 2024-09-21T02:06:29.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fe1f60 0x55c366387f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366388440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c3662ffc10 con 0x55c365fc1280 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c3662ef040 con 0x55c365fb90a0 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2580fd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c3662f2e50 con 0x55c365fe1f60 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49462/0 (socket says 172.21.15.16:49462) 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2573fff640 1 -- 172.21.15.16:0/1823240653 learned_addr learned my addr 172.21.15.16:0/1823240653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.620+0000 7f2579002640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fe1f60 0x55c366387f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2578801640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366388440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 -- 172.21.15.16:0/1823240653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fe1f60 msgr2=0x55c366387f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fe1f60 0x55c366387f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 -- 172.21.15.16:0/1823240653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 msgr2=0x55c366388440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366388440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 -- 172.21.15.16:0/1823240653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36604d540 con 0x55c365fc1280 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2578801640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366388440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2573fff640 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2560105790 tx=0x7f2560110660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad40370ff0d97e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2571ffb640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 0x7f25640b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2579002640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 0x7f25640b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f255effd640 1 -- 172.21.15.16:0/1823240653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600cd640 con 0x55c365fc1280 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f255effd640 1 -- 172.21.15.16:0/1823240653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2560115020 con 0x55c365fc1280 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f255effd640 1 -- 172.21.15.16:0/1823240653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600cd820 con 0x55c365fc1280 2024-09-21T02:06:29.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.621+0000 7f2571ffb640 1 -- 172.21.15.16:0/956654208 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2560108330 con 0x7f25741094c0 2024-09-21T02:06:29.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 -- 172.21.15.16:0/1823240653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 msgr2=0x55c366388980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2579002640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 0x7f25640b8b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f256c007e50 tx=0x7f256c007600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2560105790 tx=0x7f2560110660 comp rx=0 tx=0).stop 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 -- 172.21.15.16:0/1823240653 shutdown_connections 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366388980 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366388440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1823240653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fe1f60 0x55c366387f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 -- 172.21.15.16:0/1823240653 >> 172.21.15.16:0/1823240653 conn(0x55c365fd6350 msgr2=0x55c366385df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 -- 172.21.15.16:0/1823240653 shutdown_connections 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.622+0000 7f2580fd9740 1 -- 172.21.15.16:0/1823240653 wait complete. 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 -- start start 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 Processor -- start 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663939d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c36639f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c3661505f0 con 0x55c365fc1280 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c36628a010 con 0x55c365fe1f60 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.623+0000 7f2580fd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c3663964a0 con 0x55c365fb90a0 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2580fd9740 10 client.0 _ll_register_callbacks cb 0x55c3662e9340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49468/0 (socket says 172.21.15.16:49468) 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 -- 172.21.15.16:0/1155211200 learned_addr learned my addr 172.21.15.16:0/1155211200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2578801640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2579002640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c36639f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 -- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 msgr2=0x55c3663939d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663939d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 -- 172.21.15.16:0/1155211200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 msgr2=0x55c36639f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c36639f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 -- 172.21.15.16:0/1155211200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3662b7a00 con 0x55c365fc1280 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2578801640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2579002640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c36639f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2573fff640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25601058d0 tx=0x7f256010ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122d01ddc8f77878 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2553fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560049070 con 0x55c365fc1280 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2560047d90 con 0x55c365fc1280 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.624+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2560115020 con 0x55c365fc1280 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2560105e60 con 0x55c365fc1280 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 10 client.15309 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:29.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c3663a2330 con 0x55c365fc1280 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2560115020 con 0x55c365fc1280 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f25600ca140 con 0x55c365fc1280 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 10 client.15309 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 10 client.15309 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 20 client.15309 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365fa08e0 0x55c3663a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 4 client.15309 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.625+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c366301b60 con 0x55c365fa08e0 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.626+0000 7f2578801640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365fa08e0 0x55c3663a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.626+0000 7f2553fff640 10 client.15309 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.626+0000 7f2578801640 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365fa08e0 0x55c3663a5b30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=10466038b7950c2d server_cookie=8fd6fa639647aa59 in_seq=0 out_seq=0 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2553fff640 1 -- 172.21.15.16:0/1155211200 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c366301b60 con 0x55c365fa08e0 2024-09-21T02:06:29.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2553fff640 10 client.15309 handle_command_reply: tid=0 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 client.15309 shutdown 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 msgr2=0x55c366393f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25601058d0 tx=0x7f256010ba70 comp rx=0 tx=0).stop 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 shutdown_connections 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365fa08e0 0x55c3663a5b30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c36639f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c366393f10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2553fff640 0 client.15309 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 --2- 172.21.15.16:0/1155211200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 >> 172.21.15.16:0/1155211200 conn(0x55c365fd6350 msgr2=0x55c36638b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 shutdown_connections 2024-09-21T02:06:29.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.627+0000 7f2580fd9740 1 -- 172.21.15.16:0/1155211200 wait complete. 2024-09-21T02:06:29.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.628+0000 7f2580fd9740 20 client.15309 trim_cache size 0 max 16384 2024-09-21T02:06:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 Processor -- start 2024-09-21T02:06:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 -- start start 2024-09-21T02:06:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366392cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fc1280 0x55c36638e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a3e70 con 0x55c365fe1f60 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a4540 con 0x55c365fb90a0 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2580fd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a4c10 con 0x55c365fc1280 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49476/0 (socket says 172.21.15.16:49476) 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 -- 172.21.15.16:0/3056422750 learned_addr learned my addr 172.21.15.16:0/3056422750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2573fff640 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fc1280 0x55c36638e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2578801640 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366392cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 -- 172.21.15.16:0/3056422750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fc1280 msgr2=0x55c36638e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fc1280 0x55c36638e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 -- 172.21.15.16:0/3056422750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 msgr2=0x55c366392cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366392cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.646+0000 7f2579002640 1 -- 172.21.15.16:0/3056422750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36638a700 con 0x55c365fe1f60 2024-09-21T02:06:29.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.647+0000 7f2579002640 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f256c01e610 tx=0x7f256c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6e0fe553f368ae server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.647+0000 7f255e7fc640 1 -- 172.21.15.16:0/3056422750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c01c840 con 0x55c365fe1f60 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.647+0000 7f255e7fc640 1 -- 172.21.15.16:0/3056422750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f256c01ce70 con 0x55c365fe1f60 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.647+0000 7f255e7fc640 1 -- 172.21.15.16:0/3056422750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c00e3b0 con 0x55c365fe1f60 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- 172.21.15.16:0/3056422750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 msgr2=0x55c36638eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f256c01e610 tx=0x7f256c014000 comp rx=0 tx=0).stop 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- 172.21.15.16:0/3056422750 shutdown_connections 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fe1f60 0x55c36638eec0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fc1280 0x55c36638e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 --2- 172.21.15.16:0/3056422750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fb90a0 0x55c366392cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- 172.21.15.16:0/3056422750 >> 172.21.15.16:0/3056422750 conn(0x55c365fd6350 msgr2=0x55c3663939b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- 172.21.15.16:0/3056422750 shutdown_connections 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- 172.21.15.16:0/3056422750 wait complete. 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 -- start start 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.648+0000 7f2580fd9740 1 Processor -- start 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663ae560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c3663a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a59b0 con 0x55c365fc1280 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a52e0 con 0x55c365fe1f60 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c3663a6080 con 0x55c365fb90a0 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2580fd9740 10 client.0 _ll_register_callbacks cb 0x7f256c0074a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49488/0 (socket says 172.21.15.16:49488) 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 -- 172.21.15.16:0/2575917348 learned_addr learned my addr 172.21.15.16:0/2575917348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2578801640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2579002640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c3663a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 -- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 msgr2=0x55c3663ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 -- 172.21.15.16:0/2575917348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 msgr2=0x55c3663a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c3663a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.649+0000 7f2573fff640 1 -- 172.21.15.16:0/2575917348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3663a8690 con 0x55c365fc1280 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2573fff640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f256010bae0 tx=0x7f2560105cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf2c0626ca1c822 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560049040 con 0x55c365fc1280 2024-09-21T02:06:29.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2560035d00 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256c01ce70 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f25600cda10 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 10 client.15315 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c3663a8690 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f256c01ce70 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f2560031400 con 0x55c365fc1280 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 10 client.15315 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 10 client.15315 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 20 client.15315 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365f75c90 0x55c3664142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 4 client.15315 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:29.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.650+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "status"}) -- 0x55c3662fd610 con 0x55c365f75c90 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2578801640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365f75c90 0x55c3664142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2578801640 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365f75c90 0x55c3664142e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e5d1faab61fbb44d server_cookie=e9c818ac787b294d in_seq=0 out_seq=0 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2553fff640 10 client.15315 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2553fff640 1 -- 172.21.15.16:0/2575917348 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c3662fd610 con 0x55c365f75c90 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2553fff640 10 client.15315 handle_command_reply: tid=0 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2580fd9740 1 client.15315 shutdown 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 msgr2=0x55c3663a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.651+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f256010bae0 tx=0x7f2560105cd0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 shutdown_connections 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x55c365f75c90 0x55c3664142e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c365fe1f60 0x55c3663a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c365fc1280 0x55c3663a7a10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 --2- 172.21.15.16:0/2575917348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c365fb90a0 0x55c3663ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 >> 172.21.15.16:0/2575917348 conn(0x55c365fd6350 msgr2=0x55c3663939b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2553fff640 0 client.15315 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 shutdown_connections 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 1 -- 172.21.15.16:0/2575917348 wait complete. 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f2580fd9740 20 client.15315 trim_cache size 0 max 16384 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:29.658 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24655, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 69.818959066999994, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 139.074306775 2024-09-21T02:06:29.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 msgr2=0x7f25640b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 0x7f25640b8b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f256c007e50 tx=0x7f256c007600 comp rx=0 tx=0).stop 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 msgr2=0x7f25741034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f256002a930 tx=0x7f2560003d60 comp rx=0 tx=0).stop 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 shutdown_connections 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25640b65f0 0x7f25640b8b00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2574109e10 0x7f2574103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25741094c0 0x7f25741034c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.652+0000 7f257aa65640 1 --2- 172.21.15.16:0/956654208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2574101fe0 0x7f2574102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:29.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.653+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 >> 172.21.15.16:0/956654208 conn(0x7f2574077bc0 msgr2=0x7f2574105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:29.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.653+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 shutdown_connections 2024-09-21T02:06:29.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:29.653+0000 7f257aa65640 1 -- 172.21.15.16:0/956654208 wait complete. 2024-09-21T02:06:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:29 smithi016 ceph-mon[31245]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:30.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:29 smithi161 ceph-mon[34061]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:30.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:29 smithi123 ceph-mon[33453]: pgmap v332: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:30.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.e status 2024-09-21T02:06:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 -- 172.21.15.16:0/303932974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6410de90 msgr2=0x7fcf64110380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 --2- 172.21.15.16:0/303932974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6410de90 0x7fcf64110380 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009ca0 tx=0x7fcf5c02a8e0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 -- 172.21.15.16:0/303932974 shutdown_connections 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 --2- 172.21.15.16:0/303932974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf6410de90 0x7fcf64110380 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 --2- 172.21.15.16:0/303932974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 --2- 172.21.15.16:0/303932974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf64104a30 0x7fcf64104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 -- 172.21.15.16:0/303932974 >> 172.21.15.16:0/303932974 conn(0x7fcf64079770 msgr2=0x7fcf64079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 -- 172.21.15.16:0/303932974 shutdown_connections 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.316+0000 7fcf69e12640 1 -- 172.21.15.16:0/303932974 wait complete. 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.317+0000 7fcf69e12640 1 Processor -- start 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.317+0000 7fcf69e12640 1 -- start start 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf6410de90 0x7fcf641a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64113a20 con 0x7fcf64104a30 2024-09-21T02:06:31.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641138a0 con 0x7fcf64105380 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64113ba0 con 0x7fcf6410de90 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49516/0 (socket says 172.21.15.16:49516) 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 -- 172.21.15.16:0/1980184954 learned_addr learned my addr 172.21.15.16:0/1980184954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf6410de90 0x7fcf641a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 -- 172.21.15.16:0/1980184954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf6410de90 msgr2=0x7fcf641a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf6410de90 0x7fcf641a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 -- 172.21.15.16:0/1980184954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 msgr2=0x7fcf6419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 -- 172.21.15.16:0/1980184954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf641a2680 con 0x7fcf64104a30 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400d770 tx=0x7fcf5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf60ff9640 1 -- 172.21.15.16:0/1980184954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54017070 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf60ff9640 1 -- 172.21.15.16:0/1980184954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcf540054b0 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf641a2970 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.318+0000 7fcf60ff9640 1 -- 172.21.15.16:0/1980184954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54004c60 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.319+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a90d0 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.321+0000 7fcf70386740 1 Processor -- start 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.321+0000 7fcf70386740 1 -- start start 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.321+0000 7fcf60ff9640 1 -- 172.21.15.16:0/1980184954 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fcf54004e00 con 0x7fcf64104a30 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55956921ff60 0x5595695c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691f70a0 0x5595695c6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55956953dc10 con 0x5595691ff280 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55956952d040 con 0x55956921ff60 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf70386740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559569530e50 con 0x5595691f70a0 2024-09-21T02:06:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49518/0 (socket says 172.21.15.16:49518) 2024-09-21T02:06:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 -- 172.21.15.16:0/2572399646 learned_addr learned my addr 172.21.15.16:0/2572399646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:31.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf637fe640 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691f70a0 0x5595695c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf63fff640 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55956921ff60 0x5595695c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 -- 172.21.15.16:0/2572399646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691f70a0 msgr2=0x5595695c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691f70a0 0x5595695c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 -- 172.21.15.16:0/2572399646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55956921ff60 msgr2=0x5595695c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55956921ff60 0x5595695c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 -- 172.21.15.16:0/2572399646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55956928b540 con 0x5595691ff280 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.322+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800a970 tx=0x7fcf5800ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54fa6578b29e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf60ff9640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 0x7fcf4c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf49ffb640 1 -- 172.21.15.16:0/2572399646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58015070 con 0x5595691ff280 2024-09-21T02:06:31.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf49ffb640 1 -- 172.21.15.16:0/2572399646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcf5800cc00 con 0x5595691ff280 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf49ffb640 1 -- 172.21.15.16:0/2572399646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf580185c0 con 0x5595691ff280 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 0x7fcf4c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.323+0000 7fcf60ff9640 1 -- 172.21.15.16:0/1980184954 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fcf540e74b0 con 0x7fcf64104a30 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- 172.21.15.16:0/2572399646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 msgr2=0x5595695c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 0x7fcf4c0b8b00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c031fd0 tx=0x7fcf5c0098a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800a970 tx=0x7fcf5800ae30 comp rx=0 tx=0).stop 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- 172.21.15.16:0/2572399646 shutdown_connections 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695c6980 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691f70a0 0x5595695c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 --2- 172.21.15.16:0/2572399646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55956921ff60 0x5595695c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- 172.21.15.16:0/2572399646 >> 172.21.15.16:0/2572399646 conn(0x559569214350 msgr2=0x5595695c3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- 172.21.15.16:0/2572399646 shutdown_connections 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- 172.21.15.16:0/2572399646 wait complete. 2024-09-21T02:06:31.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 -- start start 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.324+0000 7fcf70386740 1 Processor -- start 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691ff280 0x5595695d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55956938e5f0 con 0x5595691f70a0 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5595694c8010 con 0x5595691ff280 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5595695d46e0 con 0x55956921ff60 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf70386740 10 client.0 _ll_register_callbacks cb 0x559569527340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49522/0 (socket says 172.21.15.16:49522) 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 -- 172.21.15.16:0/4286677117 learned_addr learned my addr 172.21.15.16:0/4286677117 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691ff280 0x5595695d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf63fff640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 -- 172.21.15.16:0/4286677117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 msgr2=0x5595695dda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695dda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 -- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691ff280 msgr2=0x5595695d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691ff280 0x5595695d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.325+0000 7fcf637fe640 1 -- 172.21.15.16:0/4286677117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595694f5a00 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf637fe640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcf540e4a00 tx=0x7fcf540e4e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf469e4dcda3c47 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54020050 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fcf540050f0 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf5800cc00 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf540ace20 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 10 client.15327 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5595695e0610 con 0x5595691f70a0 2024-09-21T02:06:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcf5800cc00 con 0x5595691f70a0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7fcf540981a0 con 0x5595691f70a0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 10 client.15327 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 10 client.15327 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.326+0000 7fcf70386740 20 client.15327 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691de8e0 0x5595695e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf70386740 4 client.15327 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55956953fb60 con 0x5595691de8e0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691de8e0 0x5595695e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf42ffd640 10 client.15327 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.327+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691de8e0 0x5595695e40b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e0fd57ad2d1c97ed server_cookie=aa040d58b802e0aa in_seq=0 out_seq=0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.328+0000 7fcf42ffd640 1 -- 172.21.15.16:0/4286677117 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55956953fb60 con 0x5595691de8e0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.328+0000 7fcf42ffd640 10 client.15327 handle_command_reply: tid=0 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.328+0000 7fcf70386740 1 client.15327 shutdown 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 msgr2=0x5595695d1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcf540e4a00 tx=0x7fcf540e4e50 comp rx=0 tx=0).stop 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 shutdown_connections 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691de8e0 0x5595695e40b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf42ffd640 0 client.15327 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691ff280 0x5595695d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 --2- 172.21.15.16:0/4286677117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691f70a0 0x5595695d1740 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 >> 172.21.15.16:0/4286677117 conn(0x559569214350 msgr2=0x5595695c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 shutdown_connections 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 1 -- 172.21.15.16:0/4286677117 wait complete. 2024-09-21T02:06:31.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.329+0000 7fcf70386740 20 client.15327 trim_cache size 0 max 16384 2024-09-21T02:06:31.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 Processor -- start 2024-09-21T02:06:31.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 -- start start 2024-09-21T02:06:31.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695ddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595695e23f0 con 0x5595691ff280 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5595695e2ac0 con 0x5595691f70a0 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.347+0000 7fcf70386740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5595695e3190 con 0x55956921ff60 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49536/0 (socket says 172.21.15.16:49536) 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 -- 172.21.15.16:0/245757736 learned_addr learned my addr 172.21.15.16:0/245757736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf63fff640 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695ddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf637fe640 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 -- 172.21.15.16:0/245757736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 msgr2=0x5595695ddeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695ddeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 -- 172.21.15.16:0/245757736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 msgr2=0x5595695cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 -- 172.21.15.16:0/245757736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595695d2a00 con 0x5595691ff280 2024-09-21T02:06:31.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800a970 tx=0x7fcf58006a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cfc11f8be5d1d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf497fa640 1 -- 172.21.15.16:0/245757736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58015040 con 0x5595691ff280 2024-09-21T02:06:31.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf497fa640 1 -- 172.21.15.16:0/245757736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcf5800a480 con 0x5595691ff280 2024-09-21T02:06:31.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.348+0000 7fcf497fa640 1 -- 172.21.15.16:0/245757736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5801a450 con 0x5595691ff280 2024-09-21T02:06:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- 172.21.15.16:0/245757736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 msgr2=0x5595695dd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800a970 tx=0x7fcf58006a30 comp rx=0 tx=0).stop 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- 172.21.15.16:0/245757736 shutdown_connections 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55956921ff60 0x5595695ddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595691ff280 0x5595695dd970 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 --2- 172.21.15.16:0/245757736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- 172.21.15.16:0/245757736 >> 172.21.15.16:0/245757736 conn(0x559569214350 msgr2=0x5595695cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- 172.21.15.16:0/245757736 shutdown_connections 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- 172.21.15.16:0/245757736 wait complete. 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.349+0000 7fcf70386740 1 -- start start 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 Processor -- start 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691ff280 0x5595695e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5595695e3f30 con 0x55956921ff60 2024-09-21T02:06:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5595695e3860 con 0x5595691f70a0 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5595695e4600 con 0x5595691ff280 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf70386740 10 client.0 _ll_register_callbacks cb 0x7fcf5400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf63fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49550/0 (socket says 172.21.15.16:49550) 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf63fff640 1 -- 172.21.15.16:0/1532717942 learned_addr learned my addr 172.21.15.16:0/1532717942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.350+0000 7fcf62ffd640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691ff280 0x5595695e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 -- 172.21.15.16:0/1532717942 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691ff280 msgr2=0x5595695e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691ff280 0x5595695e62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 -- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 msgr2=0x5595695ecd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695ecd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 -- 172.21.15.16:0/1532717942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595695eb9d0 con 0x55956921ff60 2024-09-21T02:06:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf63fff640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c003970 tx=0x7fcf5c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c79da5935f9741 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c058060 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fcf5c0047d0 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf5800a480 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf5c05f050 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf70386740 10 client.15333 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55956964f770 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.351+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcf5800a480 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7fcf5c053630 con 0x55956921ff60 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 10 client.15333 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 10 client.15333 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 20 client.15333 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691b3c90 0x5595696530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:31.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 4 client.15333 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "status"}) -- 0x55956953b610 con 0x5595691b3c90 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691b3c90 0x5595696530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf42ffd640 10 client.15333 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.352+0000 7fcf637fe640 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691b3c90 0x5595696530f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c0e7fe4a7a3960a server_cookie=13fffacec556c301 in_seq=0 out_seq=0 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf42ffd640 1 -- 172.21.15.16:0/1532717942 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55956953b610 con 0x5595691b3c90 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf42ffd640 10 client.15333 handle_command_reply: tid=0 2024-09-21T02:06:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 client.15333 shutdown 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 msgr2=0x5595695e69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c003970 tx=0x7fcf5c0095c0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 shutdown_connections 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5595691b3c90 0x5595696530f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55956921ff60 0x5595695e69d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5595691ff280 0x5595695e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 --2- 172.21.15.16:0/1532717942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5595691f70a0 0x5595695ecd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 >> 172.21.15.16:0/1532717942 conn(0x559569214350 msgr2=0x5595695d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf42ffd640 0 client.15333 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 shutdown_connections 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 1 -- 172.21.15.16:0/1532717942 wait complete. 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.353+0000 7fcf70386740 20 client.15333 trim_cache size 0 max 16384 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24659, 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:31.359 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 72.527587080999993, 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 146.01945424799999 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 msgr2=0x7fcf4c0b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 0x7fcf4c0b8b00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c031fd0 tx=0x7fcf5c0098a0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 msgr2=0x7fcf6419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400d770 tx=0x7fcf5400dc30 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 shutdown_connections 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcf4c0b65f0 0x7fcf4c0b8b00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf6410de90 0x7fcf641a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf64105380 0x7fcf6419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 --2- 172.21.15.16:0/1980184954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104a30 0x7fcf6419c710 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 >> 172.21.15.16:0/1980184954 conn(0x7fcf64079770 msgr2=0x7fcf64100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 shutdown_connections 2024-09-21T02:06:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:31.354+0000 7fcf69e12640 1 -- 172.21.15.16:0/1980184954 wait complete. 2024-09-21T02:06:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:31 smithi016 ceph-mon[31245]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:32.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:31 smithi161 ceph-mon[34061]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:32.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:06:32.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:31 smithi123 ceph-mon[33453]: pgmap v333: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 -- 172.21.15.16:0/2409669314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 msgr2=0x7f8ba4103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/2409669314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba4103fc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009900 tx=0x7f8b9402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 -- 172.21.15.16:0/2409669314 shutdown_connections 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/2409669314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 0x7f8ba4110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/2409669314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba4109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/2409669314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba4103fc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 -- 172.21.15.16:0/2409669314 >> 172.21.15.16:0/2409669314 conn(0x7f8ba40ff600 msgr2=0x7f8ba4101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 -- 172.21.15.16:0/2409669314 shutdown_connections 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.037+0000 7f8ba9f46640 1 -- 172.21.15.16:0/2409669314 wait complete. 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 Processor -- start 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 -- start start 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 0x7f8ba419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4113a60 con 0x7f8ba4103bb0 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba41138e0 con 0x7f8ba41094f0 2024-09-21T02:06:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.038+0000 7f8ba9f46640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4113be0 con 0x7f8ba4109e40 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49560/0 (socket says 172.21.15.16:49560) 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 -- 172.21.15.16:0/116769123 learned_addr learned my addr 172.21.15.16:0/116769123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 0x7f8ba419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 -- 172.21.15.16:0/116769123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 msgr2=0x7f8ba419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 0x7f8ba419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8b9affd640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 -- 172.21.15.16:0/116769123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 msgr2=0x7f8ba419aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba419aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 -- 172.21.15.16:0/116769123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba41a03e0 con 0x7f8ba4103bb0 2024-09-21T02:06:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8b9affd640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b9402a930 tx=0x7f8b94003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba17fa640 1 -- 172.21.15.16:0/116769123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9403f070 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba41a0670 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba17fa640 1 -- 172.21.15.16:0/116769123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b94003e40 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.039+0000 7f8ba17fa640 1 -- 172.21.15.16:0/116769123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94035030 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.040+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba41a6e30 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 Processor -- start 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 -- start start 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8ba17fa640 1 -- 172.21.15.16:0/116769123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f8b94042070 con 0x7f8ba4103bb0 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef56f60 0x55cbef2fcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 0x55cbef2fd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbef274c10 con 0x55cbeef36280 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cbef264040 con 0x55cbeef56f60 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.042+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbef267e50 con 0x55cbeef2e0a0 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49572/0 (socket says 172.21.15.16:49572) 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 -- 172.21.15.16:0/2877327723 learned_addr learned my addr 172.21.15.16:0/2877327723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef56f60 0x55cbef2fcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 0x55cbef2fd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 -- 172.21.15.16:0/2877327723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 msgr2=0x55cbef2fd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 0x55cbef2fd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 -- 172.21.15.16:0/2877327723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef56f60 msgr2=0x55cbef2fcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef56f60 0x55cbef2fcf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 -- 172.21.15.16:0/2877327723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbeefc2540 con 0x55cbeef36280 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 0x55cbef2fd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba17fa640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 0x7f8b900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b9affd640 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a970 tx=0x7f8b8c00ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b8bcbb25fbee9d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 0x7f8b900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8ba17fa640 1 -- 172.21.15.16:0/116769123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f8b941081e0 con 0x7f8ba4103bb0 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2877327723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c015070 con 0x55cbeef36280 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.043+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2877327723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b8c00cc00 con 0x55cbeef36280 2024-09-21T02:06:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.044+0000 7f8b99ffb640 1 -- 172.21.15.16:0/2877327723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0185c0 con 0x55cbeef36280 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 0x7f8b900b8b00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c005fd0 tx=0x7f8b9c009600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- 172.21.15.16:0/2877327723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 msgr2=0x55cbef2fd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a970 tx=0x7f8b8c00ae30 comp rx=0 tx=0).stop 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- 172.21.15.16:0/2877327723 shutdown_connections 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef2fd980 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef2e0a0 0x55cbef2fd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/2877327723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef56f60 0x55cbef2fcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- 172.21.15.16:0/2877327723 >> 172.21.15.16:0/2877327723 conn(0x55cbeef4b350 msgr2=0x55cbef2fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- 172.21.15.16:0/2877327723 shutdown_connections 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- 172.21.15.16:0/2877327723 wait complete. 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.045+0000 7f8bb04ba740 1 -- start start 2024-09-21T02:06:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 Processor -- start 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef3089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef314a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbef0c55f0 con 0x55cbeef36280 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cbef1ff010 con 0x55cbeef2e0a0 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbef30b6e0 con 0x55cbeef56f60 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8bb04ba740 10 client.0 _ll_register_callbacks cb 0x55cbef25e340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49578/0 (socket says 172.21.15.16:49578) 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8b9affd640 1 -- 172.21.15.16:0/1426969363 learned_addr learned my addr 172.21.15.16:0/1426969363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef314a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.046+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef3089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 -- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 msgr2=0x55cbef314a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef314a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 -- 172.21.15.16:0/1426969363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 msgr2=0x55cbef3089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef3089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 -- 172.21.15.16:0/1426969363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbef22ca00 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a120 tx=0x7f8b8c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8ba120bdd34628 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c020070 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f8b8c00ca00 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b8c00cc00 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b8c015040 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8bb04ba740 10 client.15345 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.047+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cbef317650 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b8c00cc00 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f8b8c024400 con 0x55cbeef36280 2024-09-21T02:06:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 10 client.15345 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 10 client.15345 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 20 client.15345 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeef158e0 0x55cbef31acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 4 client.15345 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbef276b60 con 0x55cbeef158e0 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.048+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeef158e0 0x55cbef31acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.049+0000 7f8b82ffd640 10 client.15345 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.049+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeef158e0 0x55cbef31acc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=238db45cca422001 server_cookie=1e5e61a1133e3dfd in_seq=0 out_seq=0 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1426969363 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cbef276b60 con 0x55cbeef158e0 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8b82ffd640 10 client.15345 handle_command_reply: tid=0 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 client.15345 shutdown 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 msgr2=0x55cbef308f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00a120 tx=0x7f8b8c011b00 comp rx=0 tx=0).stop 2024-09-21T02:06:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 shutdown_connections 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef314a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef308f30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeef158e0 0x55cbef31acc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1426969363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef3089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 >> 172.21.15.16:0/1426969363 conn(0x55cbeef4b350 msgr2=0x55cbef300090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8b82ffd640 0 client.15345 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 shutdown_connections 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1426969363 wait complete. 2024-09-21T02:06:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.050+0000 7f8bb04ba740 20 client.15345 trim_cache size 0 max 16384 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 Processor -- start 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 -- start start 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef31a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef303ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbef319000 con 0x55cbeef36280 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cbef3196d0 con 0x55cbeef2e0a0 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbef319da0 con 0x55cbeef56f60 2024-09-21T02:06:33.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49580/0 (socket says 172.21.15.16:49580) 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 -- 172.21.15.16:0/1532405280 learned_addr learned my addr 172.21.15.16:0/1532405280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef31a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 -- 172.21.15.16:0/1532405280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 msgr2=0x55cbef303ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef303ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef303ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 -- 172.21.15.16:0/1532405280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 msgr2=0x55cbef31a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef31a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8b9affd640 1 -- 172.21.15.16:0/1532405280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbef300f80 con 0x55cbeef36280 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.069+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef31a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.070+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00ae70 tx=0x7f8b8c006de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ae18a878f424d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.070+0000 7f8b997fa640 1 -- 172.21.15.16:0/1532405280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c020040 con 0x55cbeef36280 2024-09-21T02:06:33.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.070+0000 7f8b997fa640 1 -- 172.21.15.16:0/1532405280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b8c011770 con 0x55cbeef36280 2024-09-21T02:06:33.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.070+0000 7f8b997fa640 1 -- 172.21.15.16:0/1532405280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00cbf0 con 0x55cbeef36280 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1532405280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 msgr2=0x55cbef3039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00ae70 tx=0x7f8b8c006de0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1532405280 shutdown_connections 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef303ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef36280 0x55cbef3039a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1532405280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef2e0a0 0x55cbef31a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1532405280 >> 172.21.15.16:0/1532405280 conn(0x55cbeef4b350 msgr2=0x55cbef3089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1532405280 shutdown_connections 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1532405280 wait complete. 2024-09-21T02:06:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.071+0000 7f8bb04ba740 1 -- start start 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 Processor -- start 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 0x55cbef31c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef31ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cbef31ab40 con 0x55cbeef2e0a0 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cbef31a470 con 0x55cbeef36280 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cbef31b210 con 0x55cbeef56f60 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8bb04ba740 10 client.0 _ll_register_callbacks cb 0x7f8b9c009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49592/0 (socket says 172.21.15.16:49592) 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8ba37fe640 1 -- 172.21.15.16:0/1228714522 learned_addr learned my addr 172.21.15.16:0/1228714522 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 0x55cbef31c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.072+0000 7f8ba3fff640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef31ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 -- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 msgr2=0x55cbef31ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef31ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 -- 172.21.15.16:0/1228714522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 msgr2=0x55cbef31c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 0x55cbef31c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 -- 172.21.15.16:0/1228714522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbef31d410 con 0x55cbeef2e0a0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 0x55cbef31c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8ba37fe640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8b94105730 tx=0x7f8b94009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12028f3484385e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8b82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94049040 con 0x55cbeef2e0a0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f8b940c9ff0 con 0x55cbeef2e0a0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b8c011770 con 0x55cbeef2e0a0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b940ce480 con 0x55cbeef2e0a0 2024-09-21T02:06:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8bb04ba740 10 client.15351 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.073+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cbef322d60 con 0x55cbeef2e0a0 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b8c011770 con 0x55cbeef2e0a0 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f8b940d7310 con 0x55cbeef2e0a0 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 10 client.15351 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 10 client.15351 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 20 client.15351 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeeeeac90 0x55cbef38db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 4 client.15351 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "status"}) -- 0x55cbef272610 con 0x55cbeeeeac90 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.074+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeeeeac90 0x55cbef38db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8b9affd640 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeeeeac90 0x55cbef38db90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e38124f55941c951 server_cookie=ebe851c61303b95a in_seq=0 out_seq=0 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8b82ffd640 10 client.15351 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8b82ffd640 1 -- 172.21.15.16:0/1228714522 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cbef272610 con 0x55cbeeeeac90 2024-09-21T02:06:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8b82ffd640 10 client.15351 handle_command_reply: tid=0 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 client.15351 shutdown 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 msgr2=0x55cbef3234e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8b94105730 tx=0x7f8b94009260 comp rx=0 tx=0).stop 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 shutdown_connections 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cbeef56f60 0x55cbef31ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55cbeeeeac90 0x55cbef38db90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cbeef36280 0x55cbef31c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8b82ffd640 0 client.15351 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 --2- 172.21.15.16:0/1228714522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cbeef2e0a0 0x55cbef3234e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 >> 172.21.15.16:0/1228714522 conn(0x55cbeef4b350 msgr2=0x55cbef3089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 shutdown_connections 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 1 -- 172.21.15.16:0/1228714522 wait complete. 2024-09-21T02:06:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.075+0000 7f8bb04ba740 20 client.15351 trim_cache size 0 max 16384 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24667, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 75.260404930000007, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 137.285378654 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 msgr2=0x7f8b900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 0x7f8b900b8b00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c005fd0 tx=0x7f8b9c009600 comp rx=0 tx=0).stop 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 msgr2=0x7f8ba419a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b9402a930 tx=0x7f8b94003c20 comp rx=0 tx=0).stop 2024-09-21T02:06:33.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 shutdown_connections 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8b900b65f0 0x7f8b900b8b00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ba4109e40 0x7f8ba419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba41094f0 0x7f8ba419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 --2- 172.21.15.16:0/116769123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ba4103bb0 0x7f8ba419a470 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 >> 172.21.15.16:0/116769123 conn(0x7f8ba40ff600 msgr2=0x7f8ba4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 shutdown_connections 2024-09-21T02:06:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:33.076+0000 7f8ba9f46640 1 -- 172.21.15.16:0/116769123 wait complete. 2024-09-21T02:06:33.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:06:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:33 smithi016 ceph-mon[31245]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:34.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:33 smithi161 ceph-mon[34061]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:34.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:33 smithi123 ceph-mon[33453]: pgmap v334: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:34.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/4101017064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84102650 msgr2=0x7f0a84102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/4101017064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84102650 0x7f0a84102a60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009900 tx=0x7f0a8002a480 comp rx=0 tx=0).stop 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/4101017064 shutdown_connections 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/4101017064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84110200 0x7f0a84112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/4101017064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/4101017064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84102650 0x7f0a84102a60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/4101017064 >> 172.21.15.16:0/4101017064 conn(0x7f0a84079760 msgr2=0x7f0a84079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/4101017064 shutdown_connections 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/4101017064 wait complete. 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.689+0000 7f0a8c1aa640 1 Processor -- start 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 -- start start 2024-09-21T02:06:34.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 0x7f0a8419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a84115bc0 con 0x7f0a84110200 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a84115a40 con 0x7f0a84102650 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8c1aa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a84115d40 con 0x7f0a84102fa0 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49610/0 (socket says 172.21.15.16:49610) 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 -- 172.21.15.16:0/81764407 learned_addr learned my addr 172.21.15.16:0/81764407 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a89f46640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 0x7f0a8419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a89745640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8419cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 -- 172.21.15.16:0/81764407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 msgr2=0x7f0a8419cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8419cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 -- 172.21.15.16:0/81764407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 msgr2=0x7f0a8419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 0x7f0a8419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a8a747640 1 -- 172.21.15.16:0/81764407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a841a2500 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a89f46640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 0x7f0a8419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.690+0000 7f0a89745640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8419cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a8a747640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00cce0 tx=0x7f0a7c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a7affd640 1 -- 172.21.15.16:0/81764407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c00f450 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a7affd640 1 -- 172.21.15.16:0/81764407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a7c00fa80 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a841a27f0 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a7affd640 1 -- 172.21.15.16:0/81764407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c018ad0 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.691+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a841a9010 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.694+0000 7f0a9271e740 1 Processor -- start 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.694+0000 7f0a7affd640 1 -- 172.21.15.16:0/81764407 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0a7c00f5f0 con 0x7f0a84110200 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 -- start start 2024-09-21T02:06:34.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9e3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9e3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623bb95ac10 con 0x5623bb63cf60 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623bb94a040 con 0x5623bb6140a0 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a9271e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623bb94de50 con 0x5623bb61c280 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49620/0 (socket says 172.21.15.16:49620) 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a89745640 1 -- 172.21.15.16:0/1584539203 learned_addr learned my addr 172.21.15.16:0/1584539203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.695+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9e3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a7affd640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 0x7f0a700b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 -- 172.21.15.16:0/1584539203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 msgr2=0x5623bb9e3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9e3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 -- 172.21.15.16:0/1584539203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 msgr2=0x5623bb9e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9e3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 -- 172.21.15.16:0/1584539203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623bb6a8540 con 0x5623bb63cf60 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a8a747640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 0x7f0a700b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a7affd640 1 -- 172.21.15.16:0/81764407 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0a7c0e8200 con 0x7f0a84110200 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a89745640 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a740165e0 tx=0x7f0a74007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce5d97cd8bf37cc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a6ffff640 1 -- 172.21.15.16:0/1584539203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74011730 con 0x5623bb63cf60 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a6ffff640 1 -- 172.21.15.16:0/1584539203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a74010040 con 0x5623bb63cf60 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.696+0000 7f0a6ffff640 1 -- 172.21.15.16:0/1584539203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74018950 con 0x5623bb63cf60 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 -- 172.21.15.16:0/1584539203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 msgr2=0x5623bb9e2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a740165e0 tx=0x7f0a74007420 comp rx=0 tx=0).stop 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a8a747640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 0x7f0a700b8b30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0a5620 tx=0x7f0a7c07aa50 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 -- 172.21.15.16:0/1584539203 shutdown_connections 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9e3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9e3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1584539203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9e2f00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.697+0000 7f0a9271e740 1 -- 172.21.15.16:0/1584539203 >> 172.21.15.16:0/1584539203 conn(0x5623bb631350 msgr2=0x5623bb9e0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.698+0000 7f0a9271e740 1 -- 172.21.15.16:0/1584539203 shutdown_connections 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.698+0000 7f0a9271e740 1 -- 172.21.15.16:0/1584539203 wait complete. 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.698+0000 7f0a9271e740 1 -- start start 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.698+0000 7f0a9271e740 1 Processor -- start 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9eed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623bb7ab5f0 con 0x5623bb63cf60 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623bb8e5010 con 0x5623bb6140a0 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623bb9f1680 con 0x5623bb61c280 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a9271e740 10 client.0 _ll_register_callbacks cb 0x5623bb944340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49636/0 (socket says 172.21.15.16:49636) 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 -- 172.21.15.16:0/1555291212 learned_addr learned my addr 172.21.15.16:0/1555291212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 -- 172.21.15.16:0/1555291212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 msgr2=0x5623bb9eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 -- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 msgr2=0x5623bb9ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.699+0000 7f0a89745640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a89745640 1 -- 172.21.15.16:0/1555291212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623bb912a00 con 0x5623bb63cf60 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a89745640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0a740163b0 tx=0x7f0a740185f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ac8126f7f75ec0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7401e040 con 0x5623bb63cf60 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a74010040 con 0x5623bb63cf60 2024-09-21T02:06:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0a740059c0 con 0x5623bb63cf60 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a7402c630 con 0x5623bb63cf60 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a9271e740 10 client.15363 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.700+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5623bb9fd520 con 0x5623bb63cf60 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a74010040 con 0x5623bb63cf60 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f0a74024070 con 0x5623bb63cf60 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 10 client.15363 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 10 client.15363 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 20 client.15363 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb5fb8e0 0x5623bba01350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 4 client.15363 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623bb95cb60 con 0x5623bb5fb8e0 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.701+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb5fb8e0 0x5623bba01350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.702+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb5fb8e0 0x5623bba01350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e316917130f698bc server_cookie=b14f0515b0743bc7 in_seq=0 out_seq=0 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.702+0000 7f0a6d7fa640 10 client.15363 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:06:34.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.702+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1555291212 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5623bb95cb60 con 0x5623bb5fb8e0 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.702+0000 7f0a6d7fa640 10 client.15363 handle_command_reply: tid=0 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.702+0000 7f0a9271e740 1 client.15363 shutdown 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 msgr2=0x5623bb9fa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0a740163b0 tx=0x7f0a740185f0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 shutdown_connections 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb5fb8e0 0x5623bba01350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb63cf60 0x5623bb9fa970 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb61c280 0x5623bb9eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1555291212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a6d7fa640 0 client.15363 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 >> 172.21.15.16:0/1555291212 conn(0x5623bb631350 msgr2=0x5623bb9e60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 shutdown_connections 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 1 -- 172.21.15.16:0/1555291212 wait complete. 2024-09-21T02:06:34.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.703+0000 7f0a9271e740 20 client.15363 trim_cache size 0 max 16384 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 Processor -- start 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 -- start start 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 0x5623bb9fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb61c280 0x5623bb9ee700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623bb9ff690 con 0x5623bb61c280 2024-09-21T02:06:34.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623bb9ffd60 con 0x5623bb6140a0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.725+0000 7f0a9271e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623bba00430 con 0x5623bb5fb8e0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89745640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 0x5623bb9fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:50872/0 (socket says 172.21.15.16:50872) 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89745640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 0x5623bb9fa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:39288/0 (socket says 172.21.15.16:39288) 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 -- 172.21.15.16:0/131268111 learned_addr learned my addr 172.21.15.16:0/131268111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89745640 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb61c280 0x5623bb9ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 -- 172.21.15.16:0/131268111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 msgr2=0x5623bb9fa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 0x5623bb9fa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 -- 172.21.15.16:0/131268111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb61c280 msgr2=0x5623bb9ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb61c280 0x5623bb9ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 -- 172.21.15.16:0/131268111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623bb9e6f80 con 0x5623bb6140a0 2024-09-21T02:06:34.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.726+0000 7f0a89f46640 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a8002aa40 tx=0x7f0a80005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ab7fe91207dc3d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.727+0000 7f0a6f7fe640 1 -- 172.21.15.16:0/131268111 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8004a020 con 0x5623bb6140a0 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.727+0000 7f0a6f7fe640 1 -- 172.21.15.16:0/131268111 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a8003d450 con 0x5623bb6140a0 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.727+0000 7f0a6f7fe640 1 -- 172.21.15.16:0/131268111 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80035040 con 0x5623bb6140a0 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- 172.21.15.16:0/131268111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 msgr2=0x5623bb9fae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a8002aa40 tx=0x7f0a80005a20 comp rx=0 tx=0).stop 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- 172.21.15.16:0/131268111 shutdown_connections 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb61c280 0x5623bb9ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb6140a0 0x5623bb9fae20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 --2- 172.21.15.16:0/131268111 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb5fb8e0 0x5623bb9fa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- 172.21.15.16:0/131268111 >> 172.21.15.16:0/131268111 conn(0x5623bb631350 msgr2=0x5623bb9ed010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- 172.21.15.16:0/131268111 shutdown_connections 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- 172.21.15.16:0/131268111 wait complete. 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 -- start start 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.728+0000 7f0a9271e740 1 Processor -- start 2024-09-21T02:06:34.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 0x5623bba08640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb61c280 0x5623bba029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623bba011d0 con 0x5623bb5fb8e0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5623bba00b00 con 0x5623bb61c280 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5623bba018a0 con 0x5623bb6140a0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a9271e740 10 client.0 _ll_register_callbacks cb 0x7f0a80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a89f46640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 0x5623bba08640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a89745640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49648/0 (socket says 172.21.15.16:49648) 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a89745640 1 -- 172.21.15.16:0/1138406022 learned_addr learned my addr 172.21.15.16:0/1138406022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.729+0000 7f0a89745640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb61c280 0x5623bba029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 -- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb61c280 msgr2=0x5623bba029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb61c280 0x5623bba029a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 -- 172.21.15.16:0/1138406022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 msgr2=0x5623bba08100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 -- 172.21.15.16:0/1138406022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623bba08ec0 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89745640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a89f46640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 0x5623bba08640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0a8002a4c0 tx=0x7f0a80004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8349fc6ccfc20ba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80053020 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a8003d450 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0a8003da10 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.730+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a8005e050 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.731+0000 7f0a9271e740 10 client.25111 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.731+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5623bba096a0 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.731+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a8003d450 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.731+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f0a8004f780 con 0x5623bb6140a0 2024-09-21T02:06:34.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.731+0000 7f0a9271e740 10 client.25111 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a9271e740 10 client.25111 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a9271e740 20 client.25111 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb63cf60 0x5623bba77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a9271e740 4 client.25111 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "status"}) -- 0x5623bb958610 con 0x5623bb63cf60 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a89745640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb63cf60 0x5623bba77950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a6d7fa640 10 client.25111 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.732+0000 7f0a89745640 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb63cf60 0x5623bba77950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e6bcf5ff64a7fcb server_cookie=1e9ca6d902a9a6c7 in_seq=0 out_seq=0 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a6d7fa640 1 -- 172.21.15.16:0/1138406022 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5623bb958610 con 0x5623bb63cf60 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a6d7fa640 10 client.25111 handle_command_reply: tid=0 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 client.25111 shutdown 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 msgr2=0x5623bba08640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 0x5623bba08640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0a8002a4c0 tx=0x7f0a80004400 comp rx=0 tx=0).stop 2024-09-21T02:06:34.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 shutdown_connections 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5623bb63cf60 0x5623bba77950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5623bb61c280 0x5623bba029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5623bb6140a0 0x5623bba08640 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a6d7fa640 0 client.25111 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 --2- 172.21.15.16:0/1138406022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623bb5fb8e0 0x5623bba08100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.733+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 >> 172.21.15.16:0/1138406022 conn(0x5623bb631350 msgr2=0x5623bb9ecf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.734+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 shutdown_connections 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.734+0000 7f0a9271e740 1 -- 172.21.15.16:0/1138406022 wait complete. 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.734+0000 7f0a9271e740 20 client.25111 trim_cache size 0 max 16384 2024-09-21T02:06:34.743 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24683, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 77.926614259999994, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 141.52642056799999 2024-09-21T02:06:34.744 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 msgr2=0x7f0a700b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 0x7f0a700b8b30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0a5620 tx=0x7f0a7c07aa50 comp rx=0 tx=0).stop 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 msgr2=0x7f0a841a1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00cce0 tx=0x7f0a7c00a620 comp rx=0 tx=0).stop 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 shutdown_connections 2024-09-21T02:06:34.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a700b6620 0x7f0a700b8b30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a84110200 0x7f0a841a1e90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a84102fa0 0x7f0a8419cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 --2- 172.21.15.16:0/81764407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a84102650 0x7f0a8419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 >> 172.21.15.16:0/81764407 conn(0x7f0a84079760 msgr2=0x7f0a841007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 shutdown_connections 2024-09-21T02:06:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:34.735+0000 7f0a8c1aa640 1 -- 172.21.15.16:0/81764407 wait complete. 2024-09-21T02:06:34.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:34 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:06:35.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:34 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:06:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:34 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:06:35.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:06:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:35 smithi016 ceph-mon[31245]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:35 smithi161 ceph-mon[34061]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:36.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:35 smithi123 ceph-mon[33453]: pgmap v335: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:36.281 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 -- 172.21.15.16:0/2864159164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 msgr2=0x7f259410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2864159164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259410f520 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f258c009ca0 tx=0x7f258c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 -- 172.21.15.16:0/2864159164 shutdown_connections 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2864159164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259410f520 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2864159164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25941083a0 0x7f25941087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2864159164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2594102a60 0x7f2594102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 -- 172.21.15.16:0/2864159164 >> 172.21.15.16:0/2864159164 conn(0x7f2594079730 msgr2=0x7f2594079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 -- 172.21.15.16:0/2864159164 shutdown_connections 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.804+0000 7f259aa7e640 1 -- 172.21.15.16:0/2864159164 wait complete. 2024-09-21T02:06:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 Processor -- start 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 -- start start 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 0x7f259419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 0x7f259419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.805+0000 7f259aa7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2594112860 con 0x7f2594108cf0 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259aa7e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f25941126e0 con 0x7f2594102a60 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259aa7e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f25941129e0 con 0x7f25941083a0 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 -- 172.21.15.16:0/2906839254 learned_addr learned my addr 172.21.15.16:0/2906839254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f2593fff640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 0x7f259419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259881a640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 0x7f259419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 -- 172.21.15.16:0/2906839254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 msgr2=0x7f259419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 0x7f259419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 -- 172.21.15.16:0/2906839254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 msgr2=0x7f259419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 0x7f259419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 -- 172.21.15.16:0/2906839254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25941a0250 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f2593fff640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 0x7f259419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259881a640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 0x7f259419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259901b640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f258c005cc0 tx=0x7f258c032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f2591ffb640 1 -- 172.21.15.16:0/2906839254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c032e00 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f2591ffb640 1 -- 172.21.15.16:0/2906839254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f258c0044d0 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25941a04e0 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.806+0000 7f2591ffb640 1 -- 172.21.15.16:0/2906839254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c030b70 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.807+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941a6d60 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.809+0000 7f25a0ff2740 1 Processor -- start 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.809+0000 7f2591ffb640 1 -- 172.21.15.16:0/2906839254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f258c030d10 con 0x7f2594108cf0 2024-09-21T02:06:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.809+0000 7f25a0ff2740 1 -- start start 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d23440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d23980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x565205c9ac10 con 0x56520597cf60 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x565205c8a040 con 0x5652059540a0 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565205c8de50 con 0x56520595c280 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49664/0 (socket says 172.21.15.16:49664) 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 -- 172.21.15.16:0/3356757755 learned_addr learned my addr 172.21.15.16:0/3356757755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d23980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f259881a640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d23440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 -- 172.21.15.16:0/3356757755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 msgr2=0x565205d23980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d23980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 -- 172.21.15.16:0/3356757755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 msgr2=0x565205d23440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d23440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2593fff640 1 -- 172.21.15.16:0/3356757755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652059e8540 con 0x56520597cf60 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f2591ffb640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 0x7f25800b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.810+0000 7f259881a640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d23440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f2591ffb640 1 -- 172.21.15.16:0/2906839254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f258c108c40 con 0x7f2594108cf0 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f2593fff640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 0x7f25800b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f2593fff640 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f257c0103a0 tx=0x7f257c008580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2279e5c65a5a2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f258affd640 1 -- 172.21.15.16:0/3356757755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c012860 con 0x56520597cf60 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f258affd640 1 -- 172.21.15.16:0/3356757755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f257c012c40 con 0x56520597cf60 2024-09-21T02:06:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.811+0000 7f258affd640 1 -- 172.21.15.16:0/3356757755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c00e6e0 con 0x56520597cf60 2024-09-21T02:06:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3356757755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 msgr2=0x565205d22f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f2593fff640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 0x7f25800b8bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f257c01d000 tx=0x7f257c00ee00 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f257c0103a0 tx=0x7f257c008580 comp rx=0 tx=0).stop 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3356757755 shutdown_connections 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d23980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d23440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3356757755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d22f00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3356757755 >> 172.21.15.16:0/3356757755 conn(0x565205971350 msgr2=0x565205d20df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3356757755 shutdown_connections 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3356757755 wait complete. 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 -- start start 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.812+0000 7f25a0ff2740 1 Processor -- start 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d2e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520597cf60 0x565205d3a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x565205aeb5f0 con 0x56520595c280 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x565205c25010 con 0x5652059540a0 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565205d316e0 con 0x56520597cf60 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f25a0ff2740 10 client.0 _ll_register_callbacks cb 0x565205c84340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f2593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49676/0 (socket says 172.21.15.16:49676) 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f2593fff640 1 -- 172.21.15.16:0/3716511324 learned_addr learned my addr 172.21.15.16:0/3716511324 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f259881a640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d2e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.813+0000 7f259901b640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520597cf60 0x565205d3a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 -- 172.21.15.16:0/3716511324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520597cf60 msgr2=0x565205d3a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520597cf60 0x565205d3a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 -- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 msgr2=0x565205d2e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d2e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 -- 172.21.15.16:0/3716511324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565205c52a00 con 0x56520595c280 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f259881a640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d2e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:36.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2593fff640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f257c010e70 tx=0x7f257c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e68fc8f03989c5e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2573fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c012df0 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f257c012c40 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f257c00e5f0 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f257c03d050 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f25a0ff2740 10 client.15381 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565205d3d570 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.814+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f257c012c40 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f257c011980 con 0x56520595c280 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 10 client.15381 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 10 client.15381 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 20 client.15381 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56520593b8e0 0x565205d40e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 4 client.15381 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565205c9cb60 con 0x56520593b8e0 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f259881a640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56520593b8e0 0x565205d40e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f2573fff640 10 client.15381 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.815+0000 7f259881a640 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56520593b8e0 0x565205d40e30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbd4bd8d6881a562 server_cookie=2ba82cba2d80cbd8 in_seq=0 out_seq=0 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f2573fff640 1 -- 172.21.15.16:0/3716511324 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x565205c9cb60 con 0x56520593b8e0 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f2573fff640 10 client.15381 handle_command_reply: tid=0 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 client.15381 shutdown 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 msgr2=0x565205d2ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f257c010e70 tx=0x7f257c015000 comp rx=0 tx=0).stop 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 shutdown_connections 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56520593b8e0 0x565205d40e30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520597cf60 0x565205d3a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520595c280 0x565205d2ec70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/3716511324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d2e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 >> 172.21.15.16:0/3716511324 conn(0x565205971350 msgr2=0x565205d260e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.816+0000 7f2573fff640 0 client.15381 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:06:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.817+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 shutdown_connections 2024-09-21T02:06:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.817+0000 7f25a0ff2740 1 -- 172.21.15.16:0/3716511324 wait complete. 2024-09-21T02:06:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.817+0000 7f25a0ff2740 20 client.15381 trim_cache size 0 max 16384 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 Processor -- start 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 -- start start 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d3a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d2cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x565205d3f170 con 0x56520597cf60 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x565205d3f840 con 0x5652059540a0 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565205d3ff10 con 0x56520595c280 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f259901b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f259901b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49678/0 (socket says 172.21.15.16:49678) 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.835+0000 7f259901b640 1 -- 172.21.15.16:0/168426488 learned_addr learned my addr 172.21.15.16:0/168426488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f2593fff640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d2cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259881a640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d3a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 -- 172.21.15.16:0/168426488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 msgr2=0x565205d2cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d2cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 -- 172.21.15.16:0/168426488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 msgr2=0x565205d3a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d3a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 -- 172.21.15.16:0/168426488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565205d29fa0 con 0x56520597cf60 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259881a640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d3a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f259901b640 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f258c106190 tx=0x7f258c0ce130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0d98fa27398709 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f258a7fc640 1 -- 172.21.15.16:0/168426488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0ceee0 con 0x56520597cf60 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f258a7fc640 1 -- 172.21.15.16:0/168426488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f258c036040 con 0x56520597cf60 2024-09-21T02:06:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.836+0000 7f258a7fc640 1 -- 172.21.15.16:0/168426488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0d7d70 con 0x56520597cf60 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- 172.21.15.16:0/168426488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 msgr2=0x565205d2d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f258c106190 tx=0x7f258c0ce130 comp rx=0 tx=0).stop 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- 172.21.15.16:0/168426488 shutdown_connections 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56520597cf60 0x565205d2d4d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d2cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/168426488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5652059540a0 0x565205d3a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- 172.21.15.16:0/168426488 >> 172.21.15.16:0/168426488 conn(0x565205971350 msgr2=0x565205d3e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- 172.21.15.16:0/168426488 shutdown_connections 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- 172.21.15.16:0/168426488 wait complete. 2024-09-21T02:06:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.837+0000 7f25a0ff2740 1 -- start start 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 Processor -- start 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d42860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 0x565205d47850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x565205d40cb0 con 0x5652059540a0 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x565205d405e0 con 0x56520597cf60 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x565205d41380 con 0x56520595c280 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f259881a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f259881a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49694/0 (socket says 172.21.15.16:49694) 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f259881a640 1 -- 172.21.15.16:0/2572225785 learned_addr learned my addr 172.21.15.16:0/2572225785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f25a0ff2740 10 client.0 _ll_register_callbacks cb 0x7f257c008ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f2593fff640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d42860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.838+0000 7f259901b640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 0x565205d47850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 -- 172.21.15.16:0/2572225785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 msgr2=0x565205d42860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d42860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 -- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 msgr2=0x565205d47850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 0x565205d47850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 -- 172.21.15.16:0/2572225785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565205d47ec0 con 0x5652059540a0 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259901b640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 0x565205d47850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f259881a640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f258400b8c0 tx=0x7f258401ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f342a9e96e5cbc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.839+0000 7f2573fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.841+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584028030 con 0x5652059540a0 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.841+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2584020040 con 0x5652059540a0 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.841+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f258c036040 con 0x5652059540a0 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2584032050 con 0x5652059540a0 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 10 client.15387 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565205d49250 con 0x5652059540a0 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f258c036040 con 0x5652059540a0 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f258401c740 con 0x5652059540a0 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 10 client.15387 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 10 client.15387 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 20 client.15387 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x565205910c90 0x565205daee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 4 client.15387 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "status"}) -- 0x565205c98610 con 0x565205910c90 2024-09-21T02:06:36.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.842+0000 7f2593fff640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x565205910c90 0x565205daee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f2573fff640 10 client.15387 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:06:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f2593fff640 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x565205910c90 0x565205daee10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d38865e9f1b3d27 server_cookie=7a45556fd2c4c80a in_seq=0 out_seq=0 2024-09-21T02:06:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f2573fff640 1 -- 172.21.15.16:0/2572225785 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x565205c98610 con 0x565205910c90 2024-09-21T02:06:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f2573fff640 10 client.15387 handle_command_reply: tid=0 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 client.15387 shutdown 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 msgr2=0x565205d422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f258400b8c0 tx=0x7f258401ead0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 shutdown_connections 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x565205910c90 0x565205daee10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56520597cf60 0x565205d47850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56520595c280 0x565205d42860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f2573fff640 0 client.15387 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 --2- 172.21.15.16:0/2572225785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5652059540a0 0x565205d422a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 >> 172.21.15.16:0/2572225785 conn(0x565205971350 msgr2=0x565205d3e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 shutdown_connections 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.843+0000 7f25a0ff2740 1 -- 172.21.15.16:0/2572225785 wait complete. 2024-09-21T02:06:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f25a0ff2740 20 client.15387 trim_cache size 0 max 16384 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24710, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 122.195628081, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 131.39099858899999 2024-09-21T02:06:36.847 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 msgr2=0x7f25800b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 0x7f25800b8bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f257c01d000 tx=0x7f257c00ee00 comp rx=0 tx=0).stop 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 msgr2=0x7f259419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f258c005cc0 tx=0x7f258c032a10 comp rx=0 tx=0).stop 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 shutdown_connections 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f25800b66a0 0x7f25800b8bb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2594108cf0 0x7f259419fbe0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25941083a0 0x7f259419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 --2- 172.21.15.16:0/2906839254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2594102a60 0x7f259419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 >> 172.21.15.16:0/2906839254 conn(0x7f2594079730 msgr2=0x7f259406b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 shutdown_connections 2024-09-21T02:06:36.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:36.844+0000 7f259aa7e640 1 -- 172.21.15.16:0/2906839254 wait complete. 2024-09-21T02:06:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:37 smithi016 ceph-mon[31245]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:37 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:37 smithi161 ceph-mon[34061]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:37 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:37 smithi123 ceph-mon[33453]: pgmap v336: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:37 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:37.555 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T02:06:37.555 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T02:06:37.566 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:06:37.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-21T02:06:37.858 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 -- 172.21.15.16:0/4101153180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 msgr2=0x7f6000103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 --2- 172.21.15.16:0/4101153180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f6000103f70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009860 tx=0x7f5ffc02a340 comp rx=0 tx=0).stop 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 -- 172.21.15.16:0/4101153180 shutdown_connections 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 --2- 172.21.15.16:0/4101153180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 0x7f6000110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 --2- 172.21.15.16:0/4101153180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 0x7f60001098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 --2- 172.21.15.16:0/4101153180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f6000103f70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 -- 172.21.15.16:0/4101153180 >> 172.21.15.16:0/4101153180 conn(0x7f600007b850 msgr2=0x7f600007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 -- 172.21.15.16:0/4101153180 shutdown_connections 2024-09-21T02:06:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.407+0000 7f600824f640 1 -- 172.21.15.16:0/4101153180 wait complete. 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 Processor -- start 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 -- start start 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 0x7f600019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 0x7f600019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6000113960 con 0x7f6000103b60 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f60001137e0 con 0x7f6000109df0 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f600824f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6000113ae0 con 0x7f60001094a0 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.408+0000 7f6005feb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60218/0 (socket says 172.21.15.16:60218) 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 -- 172.21.15.16:0/3984441048 learned_addr learned my addr 172.21.15.16:0/3984441048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f60067ec640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 0x7f600019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f60057ea640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 0x7f600019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 -- 172.21.15.16:0/3984441048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 msgr2=0x7f600019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 0x7f600019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 -- 172.21.15.16:0/3984441048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 msgr2=0x7f600019fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 0x7f600019fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 -- 172.21.15.16:0/3984441048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60001a03a0 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f6005feb640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0057c0 tx=0x7f5ffc02ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc040020 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ffc0044d0 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60001a0630 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc037040 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.409+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60001a6df0 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.411+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6000103f70 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.412+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f5ffc043020 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.413+0000 7f5ff6ffd640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 0x7f5fec0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.413+0000 7f60057ea640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 0x7f5fec0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.413+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f5ffc107850 con 0x7f6000103b60 2024-09-21T02:06:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.413+0000 7f60057ea640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 0x7f5fec0b8b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6000106610 tx=0x7f5ff0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.417+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f5ffc04a020 con 0x7f6000103b60 2024-09-21T02:06:38.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.624+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f6000063800 con 0x7f5fec0b6670 2024-09-21T02:06:38.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.626+0000 7f5ff6ffd640 1 -- 172.21.15.16:0/3984441048 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6000063800 con 0x7f5fec0b6670 2024-09-21T02:06:38.626 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-09-21T02:06:38.627 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-09-21T02:06:38.627 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-09-21T02:06:38.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 msgr2=0x7f5fec0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:38.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 0x7f5fec0b8b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6000106610 tx=0x7f5ff0005f90 comp rx=0 tx=0).stop 2024-09-21T02:06:38.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 msgr2=0x7f600019a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:38.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0057c0 tx=0x7f5ffc02ae60 comp rx=0 tx=0).stop 2024-09-21T02:06:38.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 shutdown_connections 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5fec0b6670 0x7f5fec0b8b80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6000109df0 0x7f600019fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60001094a0 0x7f600019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 --2- 172.21.15.16:0/3984441048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6000103b60 0x7f600019a430 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 >> 172.21.15.16:0/3984441048 conn(0x7f600007b850 msgr2=0x7f6000107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 shutdown_connections 2024-09-21T02:06:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:38.628+0000 7f600824f640 1 -- 172.21.15.16:0/3984441048 wait complete. 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:38.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:38 smithi016 ceph-mon[31245]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:38 smithi161 ceph-mon[34061]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: pgmap v337: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:06:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:06:39.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:06:39.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:38 smithi123 ceph-mon[33453]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:39.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-21T02:06:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 -- 172.21.15.16:0/3007695380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 msgr2=0x7f33a4103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 --2- 172.21.15.16:0/3007695380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a4103f90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f339c009900 tx=0x7f339c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 -- 172.21.15.16:0/3007695380 shutdown_connections 2024-09-21T02:06:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 --2- 172.21.15.16:0/3007695380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 --2- 172.21.15.16:0/3007695380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 0x7f33a41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 --2- 172.21.15.16:0/3007695380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a4103f90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 -- 172.21.15.16:0/3007695380 >> 172.21.15.16:0/3007695380 conn(0x7f33a40ff610 msgr2=0x7f33a4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 -- 172.21.15.16:0/3007695380 shutdown_connections 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.097+0000 7f33a9112640 1 -- 172.21.15.16:0/3007695380 wait complete. 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.098+0000 7f33a9112640 1 Processor -- start 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.098+0000 7f33a9112640 1 -- start start 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 0x7f33a419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4113980 con 0x7f33a4103b80 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4113800 con 0x7f33a4109e10 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a9112640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4113b00 con 0x7f33a41094c0 2024-09-21T02:06:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60234/0 (socket says 172.21.15.16:60234) 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 -- 172.21.15.16:0/816420985 learned_addr learned my addr 172.21.15.16:0/816420985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a359e640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a259c640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 0x7f33a419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 -- 172.21.15.16:0/816420985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 msgr2=0x7f33a419aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 0x7f33a419aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 -- 172.21.15.16:0/816420985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 msgr2=0x7f33a419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 -- 172.21.15.16:0/816420985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a41a03e0 con 0x7f33a4103b80 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a359e640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.099+0000 7f33a2d9d640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f339c009230 tx=0x7f339c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.100+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c03f030 con 0x7f33a4103b80 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.100+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a41a0670 con 0x7f33a4103b80 2024-09-21T02:06:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.100+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f339c003a80 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.100+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c035070 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.100+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a41a6e30 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.101+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a4104e00 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.103+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f339c042030 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.104+0000 7f339bfff640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 0x7f33900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.104+0000 7f33a259c640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 0x7f33900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.104+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f339c108410 con 0x7f33a4103b80 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.104+0000 7f33a259c640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 0x7f33900b8b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f33a4108f00 tx=0x7f338c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:40.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.108+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f339c0d2a90 con 0x7f33a4103b80 2024-09-21T02:06:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.306+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f33a4063800 con 0x7f33900b65f0 2024-09-21T02:06:40.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.319+0000 7f339bfff640 1 -- 172.21.15.16:0/816420985 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f33a4063800 con 0x7f33900b65f0 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (2m) 2s ago 2m 85.5M - 19.3.0-5120-g18432218 ddce307e1f32 5cceb684e140 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi123 running (2m) 4s ago 2m 93.8M - 19.3.0-5120-g18432218 ddce307e1f32 3f34c452239a 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi161 running (2m) 4s ago 2m 87.2M - 19.3.0-5120-g18432218 ddce307e1f32 40d344ebc1c0 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (2m) 2s ago 2m 84.9M - 19.3.0-5120-g18432218 ddce307e1f32 350fd734ec19 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi123 running (2m) 4s ago 2m 87.3M - 19.3.0-5120-g18432218 ddce307e1f32 fcb7c60c4c90 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi161 running (2m) 4s ago 2m 89.1M - 19.3.0-5120-g18432218 ddce307e1f32 a15f11d79e22 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (2m) 2s ago 2m 86.4M - 19.3.0-5120-g18432218 ddce307e1f32 d2cd596694a2 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi123 running (2m) 4s ago 2m 87.3M - 19.3.0-5120-g18432218 ddce307e1f32 09b843ce12a5 2024-09-21T02:06:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi161 running (2m) 4s ago 2m 87.8M - 19.3.0-5120-g18432218 ddce307e1f32 4a6b1fc6e4c3 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (2m) 2s ago 2m 82.0M - 19.3.0-5120-g18432218 ddce307e1f32 d6606fb80496 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi123 running (2m) 4s ago 2m 85.8M - 19.3.0-5120-g18432218 ddce307e1f32 3c3f3738d16c 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi161 running (2m) 4s ago 2m 87.8M - 19.3.0-5120-g18432218 ddce307e1f32 5345a6151198 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (10m) 2s ago 10m 589M - 19.3.0-5120-g18432218 ddce307e1f32 8cd744d910c1 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi123 *:8443,8765 running (8m) 4s ago 8m 527M - 19.3.0-5120-g18432218 ddce307e1f32 caaf2d42c60b 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi161 *:8443,8765 running (8m) 4s ago 8m 525M - 19.3.0-5120-g18432218 ddce307e1f32 6870ba08b8c3 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 2s ago 10m 121M 2048M 19.3.0-5120-g18432218 ddce307e1f32 e5de3bb59457 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi123 running (9m) 4s ago 9m 109M 2048M 19.3.0-5120-g18432218 ddce307e1f32 407c11fb6f06 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi161 running (9m) 4s ago 9m 110M 2048M 19.3.0-5120-g18432218 ddce307e1f32 c6d18eb1eb68 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 2s ago 7m 124M 4273M 19.3.0-5120-g18432218 ddce307e1f32 6f5e5e73913d 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi123 running (7m) 4s ago 7m 127M 4272M 19.3.0-5120-g18432218 ddce307e1f32 bc1d5a158bc4 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi161 running (6m) 4s ago 6m 125M 4272M 19.3.0-5120-g18432218 ddce307e1f32 d201fb31ecb2 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 2s ago 6m 122M 4273M 19.3.0-5120-g18432218 ddce307e1f32 b110ba99eed9 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi123 running (6m) 4s ago 6m 112M 4272M 19.3.0-5120-g18432218 ddce307e1f32 a359f849c2e2 2024-09-21T02:06:40.321 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi161 running (5m) 4s ago 5m 123M 4272M 19.3.0-5120-g18432218 ddce307e1f32 1ce949d811d9 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (5m) 2s ago 5m 120M 4273M 19.3.0-5120-g18432218 ddce307e1f32 1d5195f746a4 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi123 running (4m) 4s ago 4m 125M 4272M 19.3.0-5120-g18432218 ddce307e1f32 3f82520d5dcb 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi161 running (4m) 4s ago 4m 112M 4272M 19.3.0-5120-g18432218 ddce307e1f32 9561e3cc1537 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (3m) 2s ago 3m 111M 4273M 19.3.0-5120-g18432218 ddce307e1f32 86da2410ffb3 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi123 running (3m) 4s ago 3m 114M 4272M 19.3.0-5120-g18432218 ddce307e1f32 0247d23cf81c 2024-09-21T02:06:40.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi161 running (3m) 4s ago 3m 120M 4272M 19.3.0-5120-g18432218 ddce307e1f32 153de7da1b5c 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 msgr2=0x7f33900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 0x7f33900b8b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f33a4108f00 tx=0x7f338c00b390 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 msgr2=0x7f33a419a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f339c009230 tx=0x7f339c004270 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 shutdown_connections 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f33900b65f0 0x7f33900b8b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33a4109e10 0x7f33a419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33a41094c0 0x7f33a419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 --2- 172.21.15.16:0/816420985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33a4103b80 0x7f33a419a470 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 >> 172.21.15.16:0/816420985 conn(0x7f33a40ff610 msgr2=0x7f33a4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 shutdown_connections 2024-09-21T02:06:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:40.321+0000 7f33a9112640 1 -- 172.21.15.16:0/816420985 wait complete. 2024-09-21T02:06:41.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-21T02:06:41.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:41 smithi016 ceph-mon[31245]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:41.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:41 smithi016 ceph-mon[31245]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:41.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:41 smithi161 ceph-mon[34061]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:41.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:41 smithi161 ceph-mon[34061]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:41 smithi123 ceph-mon[33453]: pgmap v338: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:41 smithi123 ceph-mon[33453]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 -- 172.21.15.16:0/2229535708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78101ff0 msgr2=0x7f2f78102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/2229535708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78101ff0 0x7f2f78102400 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009900 tx=0x7f2f6802a410 comp rx=0 tx=0).stop 2024-09-21T02:06:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 -- 172.21.15.16:0/2229535708 shutdown_connections 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/2229535708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78106c20 0x7f2f78110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/2229535708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 0x7f2f781066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/2229535708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78101ff0 0x7f2f78102400 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 -- 172.21.15.16:0/2229535708 >> 172.21.15.16:0/2229535708 conn(0x7f2f78079e30 msgr2=0x7f2f7807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 -- 172.21.15.16:0/2229535708 shutdown_connections 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.950+0000 7f2f7ef45640 1 -- 172.21.15.16:0/2229535708 wait complete. 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 Processor -- start 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 -- start start 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78101ff0 0x7f2f7819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 0x7f2f7819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781139c0 con 0x7f2f78106c20 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78113840 con 0x7f2f781062d0 2024-09-21T02:06:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7ef45640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78113b40 con 0x7f2f78101ff0 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7d4e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7d4e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60248/0 (socket says 172.21.15.16:60248) 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7d4e2640 1 -- 172.21.15.16:0/1405724955 learned_addr learned my addr 172.21.15.16:0/1405724955 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f7cce1640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78101ff0 0x7f2f7819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.951+0000 7f2f77fff640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 0x7f2f7819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 -- 172.21.15.16:0/1405724955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78101ff0 msgr2=0x7f2f7819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78101ff0 0x7f2f7819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 -- 172.21.15.16:0/1405724955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 msgr2=0x7f2f7819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 0x7f2f7819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 -- 172.21.15.16:0/1405724955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f781a0380 con 0x7f2f78106c20 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7d4e2640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000eb10 tx=0x7f2f7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70004b20 con 0x7f2f78106c20 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f781a0670 con 0x7f2f78106c20 2024-09-21T02:06:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f70004cc0 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70010690 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.952+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f781a6dd0 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.953+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7819b540 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.955+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2f70005000 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.956+0000 7f2f75ffb640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 0x7f2f600b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.956+0000 7f2f7cce1640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 0x7f2f600b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.956+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2f700e8160 con 0x7f2f78106c20 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.956+0000 7f2f7cce1640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 0x7f2f600b8b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f68006010 tx=0x7f2f68000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:41.959+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2f70056710 con 0x7f2f78106c20 2024-09-21T02:06:42.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.156+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f2f78063800 con 0x7f2f600b65f0 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.163+0000 7f2f75ffb640 1 -- 172.21.15.16:0/1405724955 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f2f78063800 con 0x7f2f600b65f0 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 6s ago 2m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 6s ago 8m smithi016=x;smithi123=y;smithi161=z;count:3 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 6s ago 9m smithi016:172.21.15.16=a;smithi123:172.21.15.123=b;smithi161:172.21.15.161=c;count:3 2024-09-21T02:06:42.164 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 6s ago - 2024-09-21T02:06:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 msgr2=0x7f2f600b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 0x7f2f600b8b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f68006010 tx=0x7f2f68000bc0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 msgr2=0x7f2f7819fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000eb10 tx=0x7f2f7000c2d0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 shutdown_connections 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f600b65f0 0x7f2f600b8b00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78106c20 0x7f2f7819fc50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f781062d0 0x7f2f7819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 --2- 172.21.15.16:0/1405724955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f78101ff0 0x7f2f7819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.165+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 >> 172.21.15.16:0/1405724955 conn(0x7f2f78079e30 msgr2=0x7f2f78108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.166+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 shutdown_connections 2024-09-21T02:06:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:42.166+0000 7f2f7ef45640 1 -- 172.21.15.16:0/1405724955 wait complete. 2024-09-21T02:06:42.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-21T02:06:43.145 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:43 smithi016 ceph-mon[31245]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:43.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:43 smithi016 ceph-mon[31245]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:43.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:43 smithi161 ceph-mon[34061]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:43.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:43 smithi161 ceph-mon[34061]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:43.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:43 smithi123 ceph-mon[33453]: pgmap v339: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:43.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:43 smithi123 ceph-mon[33453]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:43.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 -- 172.21.15.16:0/756239860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a50105340 msgr2=0x7f9a5010d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:43.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 --2- 172.21.15.16:0/756239860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a50105340 0x7f9a5010d890 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009900 tx=0x7f9a4402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:43.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 -- 172.21.15.16:0/756239860 shutdown_connections 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 --2- 172.21.15.16:0/756239860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a5010de50 0x7f9a50110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 --2- 172.21.15.16:0/756239860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a50105340 0x7f9a5010d890 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 --2- 172.21.15.16:0/756239860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a501049f0 0x7f9a50104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 -- 172.21.15.16:0/756239860 >> 172.21.15.16:0/756239860 conn(0x7f9a50079750 msgr2=0x7f9a50079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 -- 172.21.15.16:0/756239860 shutdown_connections 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.670+0000 7f9a563e2640 1 -- 172.21.15.16:0/756239860 wait complete. 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 Processor -- start 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 -- start start 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a50105340 0x7f9a5019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 0x7f9a5019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50113920 con 0x7f9a501049f0 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501137a0 con 0x7f9a5010de50 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a563e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50113aa0 con 0x7f9a50105340 2024-09-21T02:06:43.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a4ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60264/0 (socket says 172.21.15.16:60264) 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.671+0000 7f9a4ffff640 1 -- 172.21.15.16:0/631360656 learned_addr learned my addr 172.21.15.16:0/631360656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a50105340 0x7f9a5019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a5497f640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 0x7f9a5019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 -- 172.21.15.16:0/631360656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a50105340 msgr2=0x7f9a5019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a50105340 0x7f9a5019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 -- 172.21.15.16:0/631360656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 msgr2=0x7f9a5019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 0x7f9a5019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 -- 172.21.15.16:0/631360656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a501a0340 con 0x7f9a501049f0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a5497f640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 0x7f9a5019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4ffff640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000e9f0 tx=0x7f9a4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4000cd80 con 0x7f9a501049f0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a501a0630 con 0x7f9a501049f0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a400108c0 con 0x7f9a501049f0 2024-09-21T02:06:43.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a400187c0 con 0x7f9a501049f0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.672+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a501a6d90 con 0x7f9a501049f0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.673+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a50104e00 con 0x7f9a501049f0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.675+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f9a40010430 con 0x7f9a501049f0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.676+0000 7f9a4d7fa640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 0x7f9a380b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.676+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 0x7f9a380b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.676+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f9a400e82c0 con 0x7f9a501049f0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.676+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 0x7f9a380b8b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a44005940 tx=0x7f9a440058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:43.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.679+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9a400ae500 con 0x7f9a501049f0 2024-09-21T02:06:43.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.879+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9a50063800 con 0x7f9a380b65f0 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.880+0000 7f9a4d7fa640 1 -- 172.21.15.16:0/631360656 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f9a50063800 con 0x7f9a380b65f0 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:smithi123 172.21.15.123 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:smithi161 172.21.15.161 2024-09-21T02:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-09-21T02:06:43.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.882+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 msgr2=0x7f9a380b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.882+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 0x7f9a380b8b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a44005940 tx=0x7f9a440058d0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.882+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 msgr2=0x7f9a5019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.882+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000e9f0 tx=0x7f9a4000eeb0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 shutdown_connections 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9a380b65f0 0x7f9a380b8b00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a5010de50 0x7f9a5019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a50105340 0x7f9a5019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 --2- 172.21.15.16:0/631360656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a501049f0 0x7f9a5019a3d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 >> 172.21.15.16:0/631360656 conn(0x7f9a50079750 msgr2=0x7f9a500ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 shutdown_connections 2024-09-21T02:06:43.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:43.883+0000 7f9a563e2640 1 -- 172.21.15.16:0/631360656 wait complete. 2024-09-21T02:06:44.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-21T02:06:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:45.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:45 smithi016 ceph-mon[31245]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:45.268 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:45 smithi016 ceph-mon[31245]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 -- 172.21.15.16:0/3007791746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 msgr2=0x7f02741098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 --2- 172.21.15.16:0/3007791746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f02741098b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0268009900 tx=0x7f026802a3f0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 -- 172.21.15.16:0/3007791746 shutdown_connections 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 --2- 172.21.15.16:0/3007791746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274109df0 0x7f0274110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 --2- 172.21.15.16:0/3007791746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f02741098b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 --2- 172.21.15.16:0/3007791746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274103b60 0x7f0274103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 -- 172.21.15.16:0/3007791746 >> 172.21.15.16:0/3007791746 conn(0x7f027407b850 msgr2=0x7f027407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.361+0000 7f027918e640 1 -- 172.21.15.16:0/3007791746 shutdown_connections 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.362+0000 7f027918e640 1 -- 172.21.15.16:0/3007791746 wait complete. 2024-09-21T02:06:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.362+0000 7f027918e640 1 Processor -- start 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.362+0000 7f027918e640 1 -- start start 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274103b60 0x7f027419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274109df0 0x7f027419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0274113960 con 0x7f02741094a0 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02741137e0 con 0x7f0274103b60 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027918e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0274113ae0 con 0x7f0274109df0 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60282/0 (socket says 172.21.15.16:60282) 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 -- 172.21.15.16:0/2763916621 learned_addr learned my addr 172.21.15.16:0/2763916621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f0272d9d640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274103b60 0x7f027419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027359e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274109df0 0x7f027419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 -- 172.21.15.16:0/2763916621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274109df0 msgr2=0x7f027419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274109df0 0x7f027419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 -- 172.21.15.16:0/2763916621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274103b60 msgr2=0x7f027419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274103b60 0x7f027419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.363+0000 7f027259c640 1 -- 172.21.15.16:0/2763916621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02741a0200 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f027259c640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0268005940 tx=0x7f02680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026803f030 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0268003c90 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02741a0490 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0268035070 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.364+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02741a09c0 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.365+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0274103f70 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.367+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0268042030 con 0x7f02741094a0 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.367+0000 7f025bfff640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 0x7f025c0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:45.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.368+0000 7f0272d9d640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 0x7f025c0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:45.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.368+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0268108220 con 0x7f02741094a0 2024-09-21T02:06:45.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.368+0000 7f0272d9d640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 0x7f025c0b8bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f026400abe0 tx=0x7f026400a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:45.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.371+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f02741a09c0 con 0x7f02741094a0 2024-09-21T02:06:45.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:45 smithi161 ceph-mon[34061]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:45.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:45 smithi161 ceph-mon[34061]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:45.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:45 smithi123 ceph-mon[33453]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:45.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:45 smithi123 ceph-mon[33453]: pgmap v340: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:06:45.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.567+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f0274063800 con 0x7f025c0b66a0 2024-09-21T02:06:45.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.569+0000 7f025bfff640 1 -- 172.21.15.16:0/2763916621 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f0274063800 con 0x7f025c0b66a0 2024-09-21T02:06:45.570 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-21T02:06:45.570 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:06:45.570 INFO:teuthology.orchestra.run.smithi016.stdout:smithi123 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400YK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:06:45.570 INFO:teuthology.orchestra.run.smithi016.stdout:smithi161 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001X400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 msgr2=0x7f025c0b8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 0x7f025c0b8bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f026400abe0 tx=0x7f026400a2d0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 msgr2=0x7f027419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0268005940 tx=0x7f02680042f0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 shutdown_connections 2024-09-21T02:06:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f025c0b66a0 0x7f025c0b8bb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0274109df0 0x7f027419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02741094a0 0x7f027419a8f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 --2- 172.21.15.16:0/2763916621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0274103b60 0x7f027419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.571+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 >> 172.21.15.16:0/2763916621 conn(0x7f027407b850 msgr2=0x7f0274105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.572+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 shutdown_connections 2024-09-21T02:06:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:45.572+0000 7f027918e640 1 -- 172.21.15.16:0/2763916621 wait complete. 2024-09-21T02:06:46.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:46 smithi016 ceph-mon[31245]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:46.252 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T02:06:46.261 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:06:46.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-21T02:06:46.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:46 smithi161 ceph-mon[34061]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:46.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:46 smithi123 ceph-mon[33453]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:46.520 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:47.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.037+0000 7f2c5653e640 1 -- 172.21.15.16:0/3318124203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c501083d0 msgr2=0x7f2c501087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.037+0000 7f2c5653e640 1 --2- 172.21.15.16:0/3318124203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c501083d0 0x7f2c501087e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009900 tx=0x7f2c4402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 -- 172.21.15.16:0/3318124203 shutdown_connections 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 --2- 172.21.15.16:0/3318124203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50108d20 0x7f2c5010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 --2- 172.21.15.16:0/3318124203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c501083d0 0x7f2c501087e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 --2- 172.21.15.16:0/3318124203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c50102ae0 0x7f2c50102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 -- 172.21.15.16:0/3318124203 >> 172.21.15.16:0/3318124203 conn(0x7f2c50079760 msgr2=0x7f2c50079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 -- 172.21.15.16:0/3318124203 shutdown_connections 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 -- 172.21.15.16:0/3318124203 wait complete. 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 Processor -- start 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.038+0000 7f2c5653e640 1 -- start start 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50102ae0 0x7f2c500ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:47.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 0x7f2c500ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50112840 con 0x7f2c50108d20 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501126c0 con 0x7f2c50102ae0 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c5653e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501129c0 con 0x7f2c501083d0 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60298/0 (socket says 172.21.15.16:60298) 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 -- 172.21.15.16:0/385031564 learned_addr learned my addr 172.21.15.16:0/385031564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c4ffff640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50102ae0 0x7f2c500ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c4f7fe640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 0x7f2c500ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 -- 172.21.15.16:0/385031564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 msgr2=0x7f2c500ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:47.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 0x7f2c500ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 -- 172.21.15.16:0/385031564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50102ae0 msgr2=0x7f2c500ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50102ae0 0x7f2c500ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 -- 172.21.15.16:0/385031564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c501a85b0 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c4f7fe640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 0x7f2c500ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.039+0000 7f2c54adb640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800cce0 tx=0x7f2c4800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.040+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800f450 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.040+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c501a88a0 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.040+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c4800fa80 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.040+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48018ad0 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.040+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c501a8db0 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.041+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c50103d10 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.043+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f2c4800f5f0 con 0x7f2c50108d20 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.044+0000 7f2c4d7fa640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 0x7f2c380b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:47.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.044+0000 7f2c4ffff640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 0x7f2c380b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.044+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2c480e8230 con 0x7f2c50108d20 2024-09-21T02:06:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.044+0000 7f2c4ffff640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 0x7f2c380b8b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2c501010c0 tx=0x7f2c4000b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.047+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2c48014030 con 0x7f2c50108d20 2024-09-21T02:06:47.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.278+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2c50100970 con 0x7f2c50108d20 2024-09-21T02:06:47.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:47 smithi016 ceph-mon[31245]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:47.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.279+0000 7f2c4d7fa640 1 -- 172.21.15.16:0/385031564 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3142 (secure 0 0 0) 0x7f2c480ae560 con 0x7f2c50108d20 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:e27 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:btime 2024-09-21T02:05:20:732012+0000 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-09-21T02:06:47.281 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 27 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-21T02:04:34.359195+0000 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-21T02:05:20.732006+0000 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-09-21T02:06:47.282 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 5 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2,3,4 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24710,1=24683,2=24667,3=24659,4=24655} 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-09-21T02:06:47.283 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3,4] 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24710 members: 24655,24659,24667,24683,24710 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24710} state up:active seq 4 addr [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{0:24643} state up:standby-replay seq 1 addr [v2:172.21.15.161:6834/1279942273,v1:172.21.15.161:6835/1279942273] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.284 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{1:24683} state up:active seq 17 addr [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{1:24626} state up:standby-replay seq 1 addr [v2:172.21.15.123:6832/3641506877,v1:172.21.15.123:6833/3641506877] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{2:24667} state up:active seq 17 addr [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{2:24595} state up:standby-replay seq 1 addr [v2:172.21.15.161:6832/2556025136,v1:172.21.15.161:6833/2556025136] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{3:24659} state up:active seq 20 addr [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{3:14853} state up:standby-replay seq 1 addr [v2:172.21.15.16:6840/3323661058,v1:172.21.15.16:6841/3323661058] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{4:24655} state up:active seq 19 addr [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{4:14841} state up:standby-replay seq 1 addr [v2:172.21.15.16:6838/1623043260,v1:172.21.15.16:6839/1623043260] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:47.285 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14781} state up:standby seq 1 addr [v2:172.21.15.16:6834/3026745530,v1:172.21.15.16:6835/3026745530] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.286 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{-1:14793} state up:standby seq 1 addr [v2:172.21.15.16:6836/3790973256,v1:172.21.15.16:6837/3790973256] compat {c=[1],r=[1],i=[1fff]}] 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 msgr2=0x7f2c380b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 0x7f2c380b8b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2c501010c0 tx=0x7f2c4000b390 comp rx=0 tx=0).stop 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 msgr2=0x7f2c501000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800cce0 tx=0x7f2c4800a620 comp rx=0 tx=0).stop 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 shutdown_connections 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2c380b65f0 0x7f2c380b8b00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c50108d20 0x7f2c501000f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c501083d0 0x7f2c500ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 --2- 172.21.15.16:0/385031564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c50102ae0 0x7f2c500ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.281+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 >> 172.21.15.16:0/385031564 conn(0x7f2c50079760 msgr2=0x7f2c50101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.282+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 shutdown_connections 2024-09-21T02:06:47.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:47.282+0000 7f2c5653e640 1 -- 172.21.15.16:0/385031564 wait complete. 2024-09-21T02:06:47.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:47 smithi161 ceph-mon[34061]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:47.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:47 smithi123 ceph-mon[33453]: pgmap v341: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:47.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- bash -c 'ceph osd dump' 2024-09-21T02:06:48.208 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:48.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:48 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/385031564' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:06:48.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:48 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/385031564' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:06:48.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:48 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/385031564' entity='client.admin' cmd={"prefix": "fs dump"} : dispatch 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 -- 172.21.15.16:0/2170800861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 msgr2=0x7fd78c110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 --2- 172.21.15.16:0/2170800861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c110620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd784009ca0 tx=0x7fd78402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 -- 172.21.15.16:0/2170800861 shutdown_connections 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 --2- 172.21.15.16:0/2170800861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c110620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 --2- 172.21.15.16:0/2170800861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c1094a0 0x7fd78c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 --2- 172.21.15.16:0/2170800861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c103b60 0x7fd78c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 -- 172.21.15.16:0/2170800861 >> 172.21.15.16:0/2170800861 conn(0x7fd78c07b850 msgr2=0x7fd78c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 -- 172.21.15.16:0/2170800861 shutdown_connections 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.706+0000 7fd791ef6640 1 -- 172.21.15.16:0/2170800861 wait complete. 2024-09-21T02:06:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 Processor -- start 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 -- start start 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c103b60 0x7fd78c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c1094a0 0x7fd78c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c1139b0 con 0x7fd78c109df0 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c113830 con 0x7fd78c103b60 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd791ef6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c113b30 con 0x7fd78c1094a0 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43312/0 (socket says 172.21.15.16:43312) 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd78bfff640 1 -- 172.21.15.16:0/1568403034 learned_addr learned my addr 172.21.15.16:0/1568403034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd783fff640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c1094a0 0x7fd78c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.707+0000 7fd78b7fe640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c103b60 0x7fd78c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 -- 172.21.15.16:0/1568403034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c1094a0 msgr2=0x7fd78c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c1094a0 0x7fd78c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 -- 172.21.15.16:0/1568403034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c103b60 msgr2=0x7fd78c19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c103b60 0x7fd78c19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 -- 172.21.15.16:0/1568403034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c1a0430 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd78bfff640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd784005e30 tx=0x7fd7840093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784032d10 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c1a06c0 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd784032eb0 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784030b20 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.708+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1a6e80 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.709+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c103f70 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.711+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fd784030ce0 con 0x7fd78c109df0 2024-09-21T02:06:48.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.711+0000 7fd7897fa640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 0x7fd7780b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:48.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.711+0000 7fd78b7fe640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 0x7fd7780b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:48.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.711+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fd784108af0 con 0x7fd78c109df0 2024-09-21T02:06:48.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.712+0000 7fd78b7fe640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 0x7fd7780b8b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005fd0 tx=0x7fd77c00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:48.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.716+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd7840ced90 con 0x7fd78c109df0 2024-09-21T02:06:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.908+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fd78c10bbe0 con 0x7fd78c109df0 2024-09-21T02:06:48.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.909+0000 7fd7897fa640 1 -- 172.21.15.16:0/1568403034 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v86) ==== 56+0+6064 (secure 0 0 0) 0x7fd7840d3170 con 0x7fd78c109df0 2024-09-21T02:06:48.910 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 86 2024-09-21T02:06:48.910 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:06:48.910 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-21T01:56:00.153453+0000 2024-09-21T02:06:48.910 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-21T02:05:54.299820+0000 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 27 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-09-21T02:06:48.911 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 9 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/1246966060,v1:172.21.15.16:6803/1246966060] [v2:172.21.15.16:6804/1246966060,v1:172.21.15.16:6805/1246966060] exists,up 2f8f1230-9fd6-4eb8-8804-c04765d2734c 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 14 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6800/4233413393,v1:172.21.15.123:6801/4233413393] [v2:172.21.15.123:6802/4233413393,v1:172.21.15.123:6803/4233413393] exists,up 354cac3a-106d-4c37-a1f3-dcbd1bdf142d 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 19 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/3882488819,v1:172.21.15.161:6801/3882488819] [v2:172.21.15.161:6802/3882488819,v1:172.21.15.161:6803/3882488819] exists,up 21a8f0b2-d4d2-4fdd-9302-7475215892e4 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 27 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/2441025500,v1:172.21.15.16:6811/2441025500] [v2:172.21.15.16:6812/2441025500,v1:172.21.15.16:6813/2441025500] exists,up eb7c5f89-293f-4a90-9a26-3ba287b3f072 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6808/1713366757,v1:172.21.15.123:6809/1713366757] [v2:172.21.15.123:6810/1713366757,v1:172.21.15.123:6811/1713366757] exists,up 56d0113f-f7b9-463f-b027-41d611c683dc 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/1422110321,v1:172.21.15.161:6809/1422110321] [v2:172.21.15.161:6810/1422110321,v1:172.21.15.161:6811/1422110321] exists,up 6d1bb4b0-fc3a-488e-8b68-41360a19cef0 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 44 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/87656955,v1:172.21.15.16:6819/87656955] [v2:172.21.15.16:6820/87656955,v1:172.21.15.16:6821/87656955] exists,up 0c00ece7-8034-440b-ba71-50cc66f698a1 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 49 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6816/271400775,v1:172.21.15.123:6817/271400775] [v2:172.21.15.123:6818/271400775,v1:172.21.15.123:6819/271400775] exists,up 15352f53-b24f-4c15-8405-1971628489a7 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 54 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/3772925422,v1:172.21.15.161:6817/3772925422] [v2:172.21.15.161:6818/3772925422,v1:172.21.15.161:6819/3772925422] exists,up 71da0d14-803c-47fa-9a77-e44887e81da4 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 60 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/638537789,v1:172.21.15.16:6827/638537789] [v2:172.21.15.16:6828/638537789,v1:172.21.15.16:6829/638537789] exists,up 55a8fc26-a3a8-4a74-97d2-68218278bb36 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 65 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.123:6824/1402496387,v1:172.21.15.123:6825/1402496387] [v2:172.21.15.123:6826/1402496387,v1:172.21.15.123:6827/1402496387] exists,up b9020569-c79b-468f-a9a0-78a812618945 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 70 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/1648324302,v1:172.21.15.161:6825/1648324302] [v2:172.21.15.161:6826/1648324302,v1:172.21.15.161:6827/1648324302] exists,up fc0b0eb6-640a-41fe-b884-b6752e5f75ed 2024-09-21T02:06:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.5 [8,11] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.b [2,4] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.10 [2,1] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.1d [8,11] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.c [3,9] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.11 [4,6] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1b [4,8] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1d [5,9] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.36 [4,9] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.f [9,10] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.10 [9,8] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1b [5,8] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1f [9,8] 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3095458517 expires 2024-09-22T01:57:51.923009+0000 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/4160624590 expires 2024-09-22T01:57:51.923009+0000 2024-09-21T02:06:48.913 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2441291809 expires 2024-09-22T01:56:22.535497+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/1946001099 expires 2024-09-22T01:56:49.870093+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3677616942 expires 2024-09-22T01:56:22.535497+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/4075907232 expires 2024-09-22T01:56:22.535497+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/1610958069 expires 2024-09-22T01:56:22.535497+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2368126584 expires 2024-09-22T01:57:51.923009+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/1610958069 expires 2024-09-22T01:56:22.535497+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3423016963 expires 2024-09-22T01:56:49.870093+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3586667721 expires 2024-09-22T01:57:51.923009+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3429917716 expires 2024-09-22T01:56:49.870093+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/4160624590 expires 2024-09-22T01:57:51.923009+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3010596393 expires 2024-09-22T01:56:49.870093+0000 2024-09-21T02:06:48.914 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/1946001099 expires 2024-09-22T01:56:49.870093+0000 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 msgr2=0x7fd7780b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 0x7fd7780b8b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005fd0 tx=0x7fd77c00b390 comp rx=0 tx=0).stop 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 msgr2=0x7fd78c19fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd784005e30 tx=0x7fd7840093d0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 shutdown_connections 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd7780b65f0 0x7fd7780b8b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd78c109df0 0x7fd78c19fd00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd78c1094a0 0x7fd78c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 --2- 172.21.15.16:0/1568403034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c103b60 0x7fd78c19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 >> 172.21.15.16:0/1568403034 conn(0x7fd78c07b850 msgr2=0x7fd78c105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 shutdown_connections 2024-09-21T02:06:48.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:48.912+0000 7fd791ef6640 1 -- 172.21.15.16:0/1568403034 wait complete. 2024-09-21T02:06:49.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:49 smithi016 ceph-mon[31245]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:49.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:49 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1568403034' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:06:49.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:49 smithi161 ceph-mon[34061]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:49.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:49 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1568403034' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:06:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:49 smithi123 ceph-mon[33453]: pgmap v342: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:49 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1568403034' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T02:06:49.625 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-21T02:06:49.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:06:49.895 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.382+0000 7f0298896640 1 -- 172.21.15.16:0/3964300486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02941094a0 msgr2=0x7f02941098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.382+0000 7f0298896640 1 --2- 172.21.15.16:0/3964300486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02941094a0 0x7f02941098b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0288009900 tx=0x7f028802a410 comp rx=0 tx=0).stop 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 -- 172.21.15.16:0/3964300486 shutdown_connections 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 --2- 172.21.15.16:0/3964300486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0294109df0 0x7f0294110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 --2- 172.21.15.16:0/3964300486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02941094a0 0x7f02941098b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 --2- 172.21.15.16:0/3964300486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294103b60 0x7f0294103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 -- 172.21.15.16:0/3964300486 >> 172.21.15.16:0/3964300486 conn(0x7f029407b850 msgr2=0x7f029407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 -- 172.21.15.16:0/3964300486 shutdown_connections 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 -- 172.21.15.16:0/3964300486 wait complete. 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.383+0000 7f0298896640 1 Processor -- start 2024-09-21T02:06:50.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 -- start start 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 0x7f029419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294109df0 0x7f029419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0294113960 con 0x7f0294103b60 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02941137e0 con 0x7f02941094a0 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0298896640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0294113ae0 con 0x7f0294109df0 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43330/0 (socket says 172.21.15.16:43330) 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 -- 172.21.15.16:0/4202870816 learned_addr learned my addr 172.21.15.16:0/4202870816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0291d9b640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 0x7f029419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0292d9d640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294109df0 0x7f029419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 -- 172.21.15.16:0/4202870816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294109df0 msgr2=0x7f029419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294109df0 0x7f029419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 -- 172.21.15.16:0/4202870816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 msgr2=0x7f029419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 0x7f029419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f029259c640 1 -- 172.21.15.16:0/4202870816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02941a02e0 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.384+0000 7f0291d9b640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 0x7f029419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f029259c640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f028400d950 tx=0x7f028400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284017030 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02941a05d0 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0284004020 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840055d0 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.385+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02941a6df0 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.386+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0294103f70 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.387+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f0284009a30 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.388+0000 7f02837fe640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 0x7f027c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.388+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f02840e7a40 con 0x7f0294103b60 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.388+0000 7f0291d9b640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 0x7f027c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:50.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.389+0000 7f0291d9b640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 0x7f027c0b8b30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f02880057c0 tx=0x7f0288009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.391+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f02840adce0 con 0x7f0294103b60 2024-09-21T02:06:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.619+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f029419b560 con 0x7f0294103b60 2024-09-21T02:06:50.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.620+0000 7f02837fe640 1 -- 172.21.15.16:0/4202870816 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7f02840b20c0 con 0x7f0294103b60 2024-09-21T02:06:50.621 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:50.621 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:06:50.621 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 msgr2=0x7f027c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 0x7f027c0b8b30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f02880057c0 tx=0x7f0288009680 comp rx=0 tx=0).stop 2024-09-21T02:06:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 msgr2=0x7f029419a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:50.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f028400d950 tx=0x7f028400de10 comp rx=0 tx=0).stop 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 shutdown_connections 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f027c0b6620 0x7f027c0b8b30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0294109df0 0x7f029419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02941094a0 0x7f029419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 --2- 172.21.15.16:0/4202870816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0294103b60 0x7f029419a430 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 >> 172.21.15.16:0/4202870816 conn(0x7f029407b850 msgr2=0x7f02941055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 shutdown_connections 2024-09-21T02:06:50.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:50.623+0000 7f0298896640 1 -- 172.21.15.16:0/4202870816 wait complete. 2024-09-21T02:06:51.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:51 smithi016 ceph-mon[31245]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:51.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:51 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4202870816' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:51.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:06:51.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:51 smithi161 ceph-mon[34061]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:51.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:51 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4202870816' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:51.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:51 smithi123 ceph-mon[33453]: pgmap v343: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:06:51.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:51 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4202870816' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 -- 172.21.15.16:0/3070433417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f410de10 msgr2=0x7f12f4110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 --2- 172.21.15.16:0/3070433417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f410de10 0x7f12f4110300 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009900 tx=0x7f12e402a410 comp rx=0 tx=0).stop 2024-09-21T02:06:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 -- 172.21.15.16:0/3070433417 shutdown_connections 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 --2- 172.21.15.16:0/3070433417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f410de10 0x7f12f4110300 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 --2- 172.21.15.16:0/3070433417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f4105300 0x7f12f410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 --2- 172.21.15.16:0/3070433417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f41049b0 0x7f12f4104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 -- 172.21.15.16:0/3070433417 >> 172.21.15.16:0/3070433417 conn(0x7f12f4079730 msgr2=0x7f12f4079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 -- 172.21.15.16:0/3070433417 shutdown_connections 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.116+0000 7f12f9172640 1 -- 172.21.15.16:0/3070433417 wait complete. 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.117+0000 7f12f9172640 1 Processor -- start 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.117+0000 7f12f9172640 1 -- start start 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f41049b0 0x7f12f419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f410de10 0x7f12f419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4113940 con 0x7f12f4105300 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41137c0 con 0x7f12f41049b0 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f9172640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4113ac0 con 0x7f12f410de10 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43354/0 (socket says 172.21.15.16:43354) 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 -- 172.21.15.16:0/40852162 learned_addr learned my addr 172.21.15.16:0/40852162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f2d9d640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f41049b0 0x7f12f419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f359e640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f410de10 0x7f12f419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 -- 172.21.15.16:0/40852162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f410de10 msgr2=0x7f12f419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f410de10 0x7f12f419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 -- 172.21.15.16:0/40852162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f41049b0 msgr2=0x7f12f419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f41049b0 0x7f12f419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 -- 172.21.15.16:0/40852162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f41a02e0 con 0x7f12f4105300 2024-09-21T02:06:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12f259c640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00d740 tx=0x7f12dc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.118+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc017070 con 0x7f12f4105300 2024-09-21T02:06:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.119+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12dc00de50 con 0x7f12f4105300 2024-09-21T02:06:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.119+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f41a05d0 con 0x7f12f4105300 2024-09-21T02:06:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.119+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00f620 con 0x7f12f4105300 2024-09-21T02:06:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.119+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f41a6d30 con 0x7f12f4105300 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.120+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f4104dc0 con 0x7f12f4105300 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.121+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7f12dc01a070 con 0x7f12f4105300 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.122+0000 7f12ebfff640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 0x7f12e00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.122+0000 7f12f2d9d640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 0x7f12e00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.122+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f12dc0e7380 con 0x7f12f4105300 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.123+0000 7f12f2d9d640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 0x7f12e00b8b00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12f406bc60 tx=0x7f12ec009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.125+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f12dc0ad620 con 0x7f12f4105300 2024-09-21T02:06:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.352+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12f419b4a0 con 0x7f12f4105300 2024-09-21T02:06:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.352+0000 7f12ebfff640 1 -- 172.21.15.16:0/40852162 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11992 (secure 0 0 0) 0x7f12f419b4a0 con 0x7f12f4105300 2024-09-21T02:06:52.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:52.354 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"btime":"2024-09-21T02:05:20:732012+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:05:20.732006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24655,24659,24667,24683,24710]},"id":1}]} 2024-09-21T02:06:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 msgr2=0x7f12e00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 0x7f12e00b8b00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12f406bc60 tx=0x7f12ec009680 comp rx=0 tx=0).stop 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 msgr2=0x7f12f419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00d740 tx=0x7f12dc00dc00 comp rx=0 tx=0).stop 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 shutdown_connections 2024-09-21T02:06:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f12e00b65f0 0x7f12e00b8b00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f410de10 0x7f12f419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f4105300 0x7f12f419a910 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 --2- 172.21.15.16:0/40852162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f41049b0 0x7f12f419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 >> 172.21.15.16:0/40852162 conn(0x7f12f4079730 msgr2=0x7f12f4100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 shutdown_connections 2024-09-21T02:06:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:52.355+0000 7f12f9172640 1 -- 172.21.15.16:0/40852162 wait complete. 2024-09-21T02:06:53.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:06:53.314 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:53 smithi016 ceph-mon[31245]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:53 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/40852162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:53.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:53 smithi161 ceph-mon[34061]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:53.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:53 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/40852162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:53 smithi123 ceph-mon[33453]: pgmap v344: 129 pgs: 129 active+clean; 649 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:06:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:53 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/40852162' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:06:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 -- 172.21.15.16:0/3343292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6001049b0 msgr2=0x7fc600104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 --2- 172.21.15.16:0/3343292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6001049b0 0x7fc600104dc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009900 tx=0x7fc5f402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 -- 172.21.15.16:0/3343292 shutdown_connections 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 --2- 172.21.15.16:0/3343292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60010de10 0x7fc600110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 --2- 172.21.15.16:0/3343292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc600105300 0x7fc60010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 --2- 172.21.15.16:0/3343292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6001049b0 0x7fc600104dc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 -- 172.21.15.16:0/3343292 >> 172.21.15.16:0/3343292 conn(0x7fc600079730 msgr2=0x7fc600079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 -- 172.21.15.16:0/3343292 shutdown_connections 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.884+0000 7fc606f81640 1 -- 172.21.15.16:0/3343292 wait complete. 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 Processor -- start 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 -- start start 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6001049b0 0x7fc60019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc60010de10 0x7fc60019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc600113920 con 0x7fc600105300 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6001137a0 con 0x7fc60010de10 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc606f81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc600113aa0 con 0x7fc6001049b0 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc5fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc5fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43372/0 (socket says 172.21.15.16:43372) 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc5fffff640 1 -- 172.21.15.16:0/1396675198 learned_addr learned my addr 172.21.15.16:0/1396675198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc60551e640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc60010de10 0x7fc60019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.885+0000 7fc604d1d640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6001049b0 0x7fc60019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 -- 172.21.15.16:0/1396675198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6001049b0 msgr2=0x7fc60019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6001049b0 0x7fc60019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 -- 172.21.15.16:0/1396675198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc60010de10 msgr2=0x7fc60019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc60010de10 0x7fc60019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 -- 172.21.15.16:0/1396675198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6001a0340 con 0x7fc600105300 2024-09-21T02:06:53.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fffff640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00d740 tx=0x7fc5ec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec017070 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc5ec00de50 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6001a0630 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec00f620 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.886+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6001a6d90 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.887+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc600104dc0 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.889+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 154580+0+0 (secure 0 0 0) 0x7fc5ec01a070 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.890+0000 7fc5fdffb640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 0x7fc5f00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.890+0000 7fc604d1d640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 0x7fc5f00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.890+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fc5ec0e7300 con 0x7fc600105300 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.891+0000 7fc604d1d640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 0x7fc5f00b8b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4006010 tx=0x7fc5f4032ba0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:53.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:53.894+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc5ec0ad620 con 0x7fc600105300 2024-09-21T02:06:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.084+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc60019b500 con 0x7fc600105300 2024-09-21T02:06:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.085+0000 7fc5fdffb640 1 -- 172.21.15.16:0/1396675198 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+25198 (secure 0 0 0) 0x7fc60019b500 con 0x7fc600105300 2024-09-21T02:06:54.087 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:06:54.087 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:05:54.299820+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 msgr2=0x7fc5f00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 0x7fc5f00b8b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4006010 tx=0x7fc5f4032ba0 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 msgr2=0x7fc60019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00d740 tx=0x7fc5ec00dc00 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 shutdown_connections 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc5f00b6670 0x7fc5f00b8b80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc60010de10 0x7fc60019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc600105300 0x7fc60019a970 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 --2- 172.21.15.16:0/1396675198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6001049b0 0x7fc60019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.088+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 >> 172.21.15.16:0/1396675198 conn(0x7fc600079730 msgr2=0x7fc600100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.089+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 shutdown_connections 2024-09-21T02:06:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:54.089+0000 7fc606f81640 1 -- 172.21.15.16:0/1396675198 wait complete. 2024-09-21T02:06:54.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:54 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1396675198' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:54.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:54 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1396675198' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:54.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:54 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1396675198' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:06:54.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:04:34.359195+0000', 'modified': '2024-09-21T02:05:20.732006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24710, 'mds_1': 24683, 'mds_2': 24667, 'mds_3': 24659, 'mds_4': 24655}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14841': {'gid': 14841, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/1623043260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1623043260}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1623043260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14853': {'gid': 14853, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/3323661058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 3323661058}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 3323661058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24595': {'gid': 24595, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6833/2556025136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6832', 'nonce': 2556025136}, {'type': 'v1', 'addr': '172.21.15.161:6833', 'nonce': 2556025136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24626': {'gid': 24626, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6833/3641506877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6832', 'nonce': 3641506877}, {'type': 'v1', 'addr': '172.21.15.123:6833', 'nonce': 3641506877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24643': {'gid': 24643, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/1279942273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 1279942273}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 1279942273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24655': {'gid': 24655, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.161:6837/2373808610', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2373808610}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2373808610}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24659': {'gid': 24659, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.123:6835/2421556237', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2421556237}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2421556237}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24667': {'gid': 24667, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2696881795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2696881795}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2696881795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24683': {'gid': 24683, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/1023560595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 1023560595}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 1023560595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24710': {'gid': 24710, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/1116723582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1116723582}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1116723582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24710, 'qdb_cluster': [24655, 24659, 24667, 24683, 24710]} 2024-09-21T02:06:54.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:06:54.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:06:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:55 smithi016 ceph-mon[31245]: pgmap v345: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T02:06:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:55 smithi016 ceph-mon[31245]: mgrmap e22: x(active, since 9m), standbys: z, y 2024-09-21T02:06:55.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:55 smithi161 ceph-mon[34061]: pgmap v345: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T02:06:55.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:55 smithi161 ceph-mon[34061]: mgrmap e22: x(active, since 9m), standbys: z, y 2024-09-21T02:06:55.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:55 smithi123 ceph-mon[33453]: pgmap v345: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-21T02:06:55.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:55 smithi123 ceph-mon[33453]: mgrmap e22: x(active, since 9m), standbys: z, y 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 -- 172.21.15.16:0/4031725338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9678109e00 msgr2=0x7f9678110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 --2- 172.21.15.16:0/4031725338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9678109e00 0x7f9678110630 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9670009900 tx=0x7f967002a410 comp rx=0 tx=0).stop 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 -- 172.21.15.16:0/4031725338 shutdown_connections 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 --2- 172.21.15.16:0/4031725338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9678109e00 0x7f9678110630 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 --2- 172.21.15.16:0/4031725338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96781094b0 0x7f96781098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 --2- 172.21.15.16:0/4031725338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678101f70 0x7f9678102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 -- 172.21.15.16:0/4031725338 >> 172.21.15.16:0/4031725338 conn(0x7f9678077c60 msgr2=0x7f96780780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 -- 172.21.15.16:0/4031725338 shutdown_connections 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.596+0000 7f967f55a640 1 -- 172.21.15.16:0/4031725338 wait complete. 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 Processor -- start 2024-09-21T02:06:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 -- start start 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9678101f70 0x7f9678102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 0x7f9678103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9678113970 con 0x7f96781094b0 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96781137f0 con 0x7f9678101f70 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967f55a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9678113af0 con 0x7f9678109e00 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43402/0 (socket says 172.21.15.16:43402) 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 -- 172.21.15.16:0/1475942777 learned_addr learned my addr 172.21.15.16:0/1475942777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967d2f6640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9678101f70 0x7f9678102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967daf7640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 0x7f9678103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 -- 172.21.15.16:0/1475942777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 msgr2=0x7f9678103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 0x7f9678103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 -- 172.21.15.16:0/1475942777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9678101f70 msgr2=0x7f9678102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9678101f70 0x7f9678102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967caf5640 1 -- 172.21.15.16:0/1475942777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96781a86a0 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.597+0000 7f967daf7640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 0x7f9678103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f967caf5640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c970 tx=0x7f966c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f96767fc640 1 -- 172.21.15.16:0/1475942777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c007bb0 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f96767fc640 1 -- 172.21.15.16:0/1475942777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f966c007d50 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f96767fc640 1 -- 172.21.15.16:0/1475942777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c004da0 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96781a8990 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.598+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96781a8f20 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 Processor -- start 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f96767fc640 1 -- 172.21.15.16:0/1475942777 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f966c004f80 con 0x7f96781094b0 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 -- start start 2024-09-21T02:06:55.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c1733c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1733c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c172b3c10 con 0x559c16f95f60 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c172a3040 con 0x559c16f75280 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f9685ace740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c172a6e50 con 0x559c16f6d0a0 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.601+0000 7f967daf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43416/0 (socket says 172.21.15.16:43416) 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 -- 172.21.15.16:0/3241014118 learned_addr learned my addr 172.21.15.16:0/3241014118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c1733c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967caf5640 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1733c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 -- 172.21.15.16:0/3241014118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 msgr2=0x559c1733c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c1733c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 -- 172.21.15.16:0/3241014118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 msgr2=0x559c1733c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1733c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 -- 172.21.15.16:0/3241014118 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c17001540 con 0x559c16f95f60 2024-09-21T02:06:55.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f96767fc640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 0x7f96600b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967daf7640 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9670009900 tx=0x7f9670032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf2efa88eddd5bc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f967d2f6640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 0x7f96600b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f96767fc640 1 -- 172.21.15.16:0/1475942777 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f966c0e7520 con 0x7f96781094b0 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.602+0000 7f965f7fe640 1 -- 172.21.15.16:0/3241014118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967003f070 con 0x559c16f95f60 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.603+0000 7f965f7fe640 1 -- 172.21.15.16:0/3241014118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96700044c0 con 0x559c16f95f60 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.603+0000 7f965f7fe640 1 -- 172.21.15.16:0/3241014118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670035070 con 0x559c16f95f60 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.603+0000 7f967d2f6640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 0x7f96600b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x559c1733ab80 tx=0x7f9668009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.603+0000 7f9685ace740 1 -- 172.21.15.16:0/3241014118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 msgr2=0x559c1733bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.603+0000 7f9685ace740 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9670009900 tx=0x7f9670032d20 comp rx=0 tx=0).stop 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 -- 172.21.15.16:0/3241014118 shutdown_connections 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1733c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c1733c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 --2- 172.21.15.16:0/3241014118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f95f60 0x559c1733bf00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 -- 172.21.15.16:0/3241014118 >> 172.21.15.16:0/3241014118 conn(0x559c16f8a350 msgr2=0x559c17339df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 -- 172.21.15.16:0/3241014118 shutdown_connections 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 -- 172.21.15.16:0/3241014118 wait complete. 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 -- start start 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.604+0000 7f9685ace740 1 Processor -- start 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c173479e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f95f60 0x559c173539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c171045f0 con 0x559c16f75280 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c1723e010 con 0x559c16f95f60 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c1734a680 con 0x559c16f6d0a0 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f9685ace740 10 client.0 _ll_register_callbacks cb 0x559c1729d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43432/0 (socket says 172.21.15.16:43432) 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 -- 172.21.15.16:0/3299442060 learned_addr learned my addr 172.21.15.16:0/3299442060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967daf7640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f95f60 0x559c173539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c173479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 -- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 msgr2=0x559c173479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c173479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 -- 172.21.15.16:0/3299442060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f95f60 msgr2=0x559c173539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f95f60 0x559c173539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 -- 172.21.15.16:0/3299442060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c1726ba00 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c173479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f967caf5640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f966c0aa430 tx=0x7f966c0a98d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1980a8e3b59dd9b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.605+0000 7f965cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c01d040 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f966c0e70f0 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559c172b5b60 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f966c0acb10 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 10 client.15459 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c17356630 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x559c172b5b60 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f966c0981a0 con 0x559c16f75280 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 10 client.15459 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 10 client.15459 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:06:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 20 client.15459 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f548e0 0x559c1735a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 4 client.15459 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.606+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c172b1610 con 0x559c16f548e0 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.607+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f548e0 0x559c1735a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.607+0000 7f965cff9640 10 client.15459 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.607+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f548e0 0x559c1735a000 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5e636ee8aa80b1c5 server_cookie=4862e16f85f28994 in_seq=0 out_seq=0 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f965cff9640 1 -- 172.21.15.16:0/3299442060 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559c172b1610 con 0x559c16f548e0 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f965cff9640 10 client.15459 handle_command_reply: tid=0 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 client.15459 shutdown 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 msgr2=0x559c17347f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f966c0aa430 tx=0x7f966c0a98d0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 shutdown_connections 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f548e0 0x559c1735a000 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f95f60 0x559c173539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f75280 0x559c17347f20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f965cff9640 0 client.15459 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 --2- 172.21.15.16:0/3299442060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c173479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 >> 172.21.15.16:0/3299442060 conn(0x559c16f8a350 msgr2=0x559c1733f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 shutdown_connections 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 1 -- 172.21.15.16:0/3299442060 wait complete. 2024-09-21T02:06:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.608+0000 7f9685ace740 20 client.15459 trim_cache size 0 max 16384 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 Processor -- start 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 -- start start 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c17353e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c17346060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c17358340 con 0x559c16f548e0 2024-09-21T02:06:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c17358a10 con 0x559c16f75280 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f9685ace740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c173590e0 con 0x559c16f6d0a0 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 -- 172.21.15.16:0/3694657567 learned_addr learned my addr 172.21.15.16:0/3694657567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967caf5640 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c17353e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967daf7640 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c17346060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 -- 172.21.15.16:0/3694657567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 msgr2=0x559c17353e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c17353e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 -- 172.21.15.16:0/3694657567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 msgr2=0x559c17346060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c17346060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.628+0000 7f967d2f6640 1 -- 172.21.15.16:0/3694657567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c1733ed60 con 0x559c16f548e0 2024-09-21T02:06:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.629+0000 7f967d2f6640 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9668005fd0 tx=0x7f9668004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4f93d547a91609 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.629+0000 7f965effd640 1 -- 172.21.15.16:0/3694657567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668004b20 con 0x559c16f548e0 2024-09-21T02:06:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.629+0000 7f965effd640 1 -- 172.21.15.16:0/3694657567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9668004cc0 con 0x559c16f548e0 2024-09-21T02:06:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.629+0000 7f965effd640 1 -- 172.21.15.16:0/3694657567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668005020 con 0x559c16f548e0 2024-09-21T02:06:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- 172.21.15.16:0/3694657567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 msgr2=0x559c17353940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9668005fd0 tx=0x7f9668004a40 comp rx=0 tx=0).stop 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- 172.21.15.16:0/3694657567 shutdown_connections 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c17346060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f6d0a0 0x559c17353e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 --2- 172.21.15.16:0/3694657567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f548e0 0x559c17353940 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- 172.21.15.16:0/3694657567 >> 172.21.15.16:0/3694657567 conn(0x559c16f8a350 msgr2=0x559c173429c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- 172.21.15.16:0/3694657567 shutdown_connections 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- 172.21.15.16:0/3694657567 wait complete. 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 -- start start 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.630+0000 7f9685ace740 1 Processor -- start 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f548e0 0x559c173628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1735c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559c17359e80 con 0x559c16f6d0a0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x559c173597b0 con 0x559c16f75280 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x559c1735a550 con 0x559c16f548e0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f9685ace740 10 client.0 _ll_register_callbacks cb 0x7f966c007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43458/0 (socket says 172.21.15.16:43458) 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 -- 172.21.15.16:0/2772253807 learned_addr learned my addr 172.21.15.16:0/2772253807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967d2f6640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f548e0 0x559c173628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967daf7640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1735c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 -- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f548e0 msgr2=0x559c173628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f548e0 0x559c173628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 -- 172.21.15.16:0/2772253807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 msgr2=0x559c1735c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1735c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 -- 172.21.15.16:0/2772253807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c1735c870 con 0x559c16f6d0a0 2024-09-21T02:06:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967daf7640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1735c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f967caf5640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f966c0986c0 tx=0x7f966c0a8d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d296b4d0fe1a9fea server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.631+0000 7f965cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c01d040 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f966c0ac0a0 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9668004cc0 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f966c0ac930 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 10 client.15465 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c173622a0 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9668004cc0 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f966c0d2c10 con 0x559c16f6d0a0 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 10 client.15465 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 20 client.15465 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f95f60 0x559c173c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 4 client.15465 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.632+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "status"}) -- 0x559c172aea40 con 0x559c16f95f60 2024-09-21T02:06:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f967d2f6640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f95f60 0x559c173c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f967d2f6640 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f95f60 0x559c173c8bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1e1255f6934a9ee6 server_cookie=ba1de3631802715d in_seq=0 out_seq=0 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f965cff9640 10 client.15465 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f965cff9640 1 -- 172.21.15.16:0/2772253807 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559c172aea40 con 0x559c16f95f60 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f965cff9640 10 client.15465 handle_command_reply: tid=0 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f9685ace740 1 client.15465 shutdown 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 msgr2=0x559c1735bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f966c0986c0 tx=0x7f966c0a8d40 comp rx=0 tx=0).stop 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.633+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 shutdown_connections 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x559c16f95f60 0x559c173c8bb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x559c16f75280 0x559c1735c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c16f6d0a0 0x559c1735bd70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f965cff9640 0 client.15465 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 --2- 172.21.15.16:0/2772253807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x559c16f548e0 0x559c173628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 >> 172.21.15.16:0/2772253807 conn(0x559c16f8a350 msgr2=0x559c17346020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 shutdown_connections 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 1 -- 172.21.15.16:0/2772253807 wait complete. 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f9685ace740 20 client.15465 trim_cache size 0 max 16384 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:55.640 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24655, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 95.801940091000006, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 165.057286941 2024-09-21T02:06:55.641 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 msgr2=0x7f96600b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 0x7f96600b8af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x559c1733ab80 tx=0x7f9668009500 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 msgr2=0x7f9678103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.634+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c970 tx=0x7f966c00ce30 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 shutdown_connections 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96600b65e0 0x7f96600b8af0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9678109e00 0x7f9678103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96781094b0 0x7f9678103450 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 --2- 172.21.15.16:0/1475942777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9678101f70 0x7f9678102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 >> 172.21.15.16:0/1475942777 conn(0x7f9678077c60 msgr2=0x7f96781054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:55.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 shutdown_connections 2024-09-21T02:06:55.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:55.635+0000 7f967f55a640 1 -- 172.21.15.16:0/1475942777 wait complete. 2024-09-21T02:06:56.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.e status 2024-09-21T02:06:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 -- 172.21.15.16:0/3528639220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c1094a0 msgr2=0x7f0c3c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3528639220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c1098b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009900 tx=0x7f0c3002a410 comp rx=0 tx=0).stop 2024-09-21T02:06:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 -- 172.21.15.16:0/3528639220 shutdown_connections 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3528639220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c109df0 0x7f0c3c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3528639220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c1098b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3528639220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 0x7f0c3c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 -- 172.21.15.16:0/3528639220 >> 172.21.15.16:0/3528639220 conn(0x7f0c3c07b850 msgr2=0x7f0c3c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 -- 172.21.15.16:0/3528639220 shutdown_connections 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.155+0000 7f0c4225a640 1 -- 172.21.15.16:0/3528639220 wait complete. 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 Processor -- start 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 -- start start 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 0x7f0c3c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c113960 con 0x7f0c3c109df0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c1137e0 con 0x7f0c3c1094a0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c4225a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c113ae0 con 0x7f0c3c103b60 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 0x7f0c3c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c3bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43482/0 (socket says 172.21.15.16:43482) 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c3bfff640 1 -- 172.21.15.16:0/3146772016 learned_addr learned my addr 172.21.15.16:0/3146772016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.156+0000 7f0c3affd640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 -- 172.21.15.16:0/3146772016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 msgr2=0x7f0c3c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 0x7f0c3c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 -- 172.21.15.16:0/3146772016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c1094a0 msgr2=0x7f0c3c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 -- 172.21.15.16:0/3146772016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3c1a0320 con 0x7f0c3c109df0 2024-09-21T02:06:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.157+0000 7f0c3bfff640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eaa0 tx=0x7f0c3400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.159+0000 7f0c38ff9640 1 -- 172.21.15.16:0/3146772016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400cdb0 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.159+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3c1a0610 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.159+0000 7f0c38ff9640 1 -- 172.21.15.16:0/3146772016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c34005020 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.159+0000 7f0c38ff9640 1 -- 172.21.15.16:0/3146772016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c340116d0 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.159+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c3c1a6d70 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.162+0000 7f0c38ff9640 1 -- 172.21.15.16:0/3146772016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f0c34011870 con 0x7f0c3c109df0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.162+0000 7f0c487ce740 1 Processor -- start 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.162+0000 7f0c487ce740 1 -- start start 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178be3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017881d280 0x560178be4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560178b5bc10 con 0x5601788150a0 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560178b4b040 con 0x56017883df60 2024-09-21T02:06:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c487ce740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560178b4ee50 con 0x56017881d280 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c38ff9640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 0x7f0c240b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43484/0 (socket says 172.21.15.16:43484) 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/4136947601 learned_addr learned my addr 172.21.15.16:0/4136947601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017881d280 0x560178be4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3bfff640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 0x7f0c240b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178be3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c38ff9640 1 -- 172.21.15.16:0/3146772016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0c340e79f0 con 0x7f0c3c109df0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/4136947601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017881d280 msgr2=0x560178be4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017881d280 0x560178be4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/4136947601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 msgr2=0x560178be3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178be3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/4136947601 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601788a9540 con 0x5601788150a0 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178be3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:06:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.163+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00bd10 tx=0x7f0c2c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130f59eedd75a14a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c3bfff640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 0x7f0c240b8a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0c340a9d10 tx=0x7f0c3407b8a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c21ffb640 1 -- 172.21.15.16:0/4136947601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c004030 con 0x5601788150a0 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c21ffb640 1 -- 172.21.15.16:0/4136947601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c2c0035f0 con 0x5601788150a0 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c21ffb640 1 -- 172.21.15.16:0/4136947601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c011640 con 0x5601788150a0 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c487ce740 1 -- 172.21.15.16:0/4136947601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 msgr2=0x560178be4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.164+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c00bd10 tx=0x7f0c2c0107a0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 -- 172.21.15.16:0/4136947601 shutdown_connections 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017881d280 0x560178be4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178be4440 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4136947601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178be3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 -- 172.21.15.16:0/4136947601 >> 172.21.15.16:0/4136947601 conn(0x560178832350 msgr2=0x560178be1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 -- 172.21.15.16:0/4136947601 shutdown_connections 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 -- 172.21.15.16:0/4136947601 wait complete. 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 -- start start 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.165+0000 7f0c487ce740 1 Processor -- start 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178befa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178beff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5601789ac5f0 con 0x56017883df60 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560178ae6010 con 0x56017881d280 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560178bf26e0 con 0x5601788150a0 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c487ce740 10 client.0 _ll_register_callbacks cb 0x560178b45340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43492/0 (socket says 172.21.15.16:43492) 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 -- 172.21.15.16:0/4239777502 learned_addr learned my addr 172.21.15.16:0/4239777502 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178befa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178beff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 -- 172.21.15.16:0/4239777502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 msgr2=0x560178befa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178befa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 -- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 msgr2=0x560178beff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178beff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 -- 172.21.15.16:0/4239777502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560178b13a00 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c3affd640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x560178bf2090 tx=0x7f0c30009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fea6b0e2e8cc6ad server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.166+0000 7f0c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30035040 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0c300035f0 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c2c0035f0 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c30054050 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 10 client.15477 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560178bfe620 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c2c0035f0 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f0c3003d660 con 0x56017883df60 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 10 client.15477 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 20 client.15477 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787fc8e0 0x560178c02060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 4 client.15477 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.167+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560178b5db60 con 0x5601787fc8e0 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.168+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787fc8e0 0x560178c02060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.168+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787fc8e0 0x560178c02060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=610b8323f170be8c server_cookie=532da75267d2f144 in_seq=0 out_seq=0 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.168+0000 7f0c1affd640 10 client.15477 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c1affd640 1 -- 172.21.15.16:0/4239777502 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560178b5db60 con 0x5601787fc8e0 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c1affd640 10 client.15477 handle_command_reply: tid=0 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 client.15477 shutdown 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 msgr2=0x560178bfb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x560178bf2090 tx=0x7f0c30009260 comp rx=0 tx=0).stop 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 shutdown_connections 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787fc8e0 0x560178c02060 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017883df60 0x560178bfb9d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c1affd640 0 client.15477 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178beff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 --2- 172.21.15.16:0/4239777502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178befa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 >> 172.21.15.16:0/4239777502 conn(0x560178832350 msgr2=0x560178be70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 shutdown_connections 2024-09-21T02:06:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 1 -- 172.21.15.16:0/4239777502 wait complete. 2024-09-21T02:06:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.169+0000 7f0c487ce740 20 client.15477 trim_cache size 0 max 16384 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 Processor -- start 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 -- start start 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178bfb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178bea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560178c003a0 con 0x56017881d280 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560178c00a70 con 0x56017883df60 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c487ce740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560178c01140 con 0x5601788150a0 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.188+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43506/0 (socket says 172.21.15.16:43506) 2024-09-21T02:06:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/214497766 learned_addr learned my addr 172.21.15.16:0/214497766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3affd640 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178bfb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3affd640 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178bea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/214497766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 msgr2=0x560178bfb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178bfb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/214497766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 msgr2=0x560178bea9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178bea9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 -- 172.21.15.16:0/214497766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560178bf20c0 con 0x56017881d280 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c011010 tx=0x7f0c2c003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed088c2be46d6d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c217fa640 1 -- 172.21.15.16:0/214497766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c005240 con 0x56017881d280 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c217fa640 1 -- 172.21.15.16:0/214497766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c2c00b8e0 con 0x56017881d280 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.189+0000 7f0c217fa640 1 -- 172.21.15.16:0/214497766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0116a0 con 0x56017881d280 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 -- 172.21.15.16:0/214497766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 msgr2=0x560178bfbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c011010 tx=0x7f0c2c003e00 comp rx=0 tx=0).stop 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 -- 172.21.15.16:0/214497766 shutdown_connections 2024-09-21T02:06:57.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017883df60 0x560178bea9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56017881d280 0x560178bfbe80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 --2- 172.21.15.16:0/214497766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5601788150a0 0x560178bfb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 -- 172.21.15.16:0/214497766 >> 172.21.15.16:0/214497766 conn(0x560178832350 msgr2=0x560178bdcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 -- 172.21.15.16:0/214497766 shutdown_connections 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.190+0000 7f0c487ce740 1 -- 172.21.15.16:0/214497766 wait complete. 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 -- start start 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 Processor -- start 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178c041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017883df60 0x560178c048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560178c01ee0 con 0x5601788150a0 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560178c01810 con 0x56017881d280 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x560178c025b0 con 0x56017883df60 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c487ce740 10 client.0 _ll_register_callbacks cb 0x7f0c30009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43518/0 (socket says 172.21.15.16:43518) 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c3affd640 1 -- 172.21.15.16:0/637647680 learned_addr learned my addr 172.21.15.16:0/637647680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.191+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178c041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017883df60 0x560178c048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 -- 172.21.15.16:0/637647680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017883df60 msgr2=0x560178c048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017883df60 0x560178c048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 -- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 msgr2=0x560178c041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178c041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 -- 172.21.15.16:0/637647680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560178c09910 con 0x5601788150a0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c3affd640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009900 tx=0x7f0c30009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796c14e467f0bc58 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30061060 con 0x5601788150a0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0c3004e110 con 0x5601788150a0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c2c00b8e0 con 0x5601788150a0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c30035040 con 0x5601788150a0 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c487ce740 10 client.15483 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:57.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.192+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560178c6d6b0 con 0x5601788150a0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c2c00b8e0 con 0x5601788150a0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f0c30031c10 con 0x5601788150a0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 10 client.15483 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 20 client.15483 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787d1c90 0x560178c70e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 4 client.15483 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "status"}) -- 0x560178b59610 con 0x5601787d1c90 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.193+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787d1c90 0x560178c70e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c1affd640 10 client.15483 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c3b7fe640 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787d1c90 0x560178c70e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b05f941d646eab22 server_cookie=7a60b5b45cf1a973 in_seq=0 out_seq=0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c1affd640 1 -- 172.21.15.16:0/637647680 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560178b59610 con 0x5601787d1c90 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c1affd640 10 client.15483 handle_command_reply: tid=0 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 client.15483 shutdown 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 msgr2=0x560178c0acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009900 tx=0x7f0c30009260 comp rx=0 tx=0).stop 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 shutdown_connections 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x5601787d1c90 0x560178c70e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56017883df60 0x560178c048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56017881d280 0x560178c041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c1affd640 0 client.15483 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 --2- 172.21.15.16:0/637647680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5601788150a0 0x560178c0acd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 >> 172.21.15.16:0/637647680 conn(0x560178832350 msgr2=0x560178bea9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 shutdown_connections 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 1 -- 172.21.15.16:0/637647680 wait complete. 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.194+0000 7f0c487ce740 20 client.15483 trim_cache size 0 max 16384 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:57.207 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24659, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 98.369698408000005, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 171.86156680799999 2024-09-21T02:06:57.208 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 msgr2=0x7f0c240b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 0x7f0c240b8a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0c340a9d10 tx=0x7f0c3407b8a0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 msgr2=0x7f0c3c19fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eaa0 tx=0x7f0c3400ef60 comp rx=0 tx=0).stop 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 shutdown_connections 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0c240b6560 0x7f0c240b8a70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3c109df0 0x7f0c3c19fbf0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c3c1094a0 0x7f0c3c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 --2- 172.21.15.16:0/3146772016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c103b60 0x7f0c3c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 >> 172.21.15.16:0/3146772016 conn(0x7f0c3c07b850 msgr2=0x7f0c3c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 shutdown_connections 2024-09-21T02:06:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:57.195+0000 7f0c4225a640 1 -- 172.21.15.16:0/3146772016 wait complete. 2024-09-21T02:06:57.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:57 smithi016 ceph-mon[31245]: pgmap v346: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:06:57.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:57 smithi161 ceph-mon[34061]: pgmap v346: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:06:57.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:57 smithi123 ceph-mon[33453]: pgmap v346: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 10 op/s 2024-09-21T02:06:57.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:06:58.201 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/2786541362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 msgr2=0x7f1b581087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/2786541362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b581087e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009900 tx=0x7f1b4c02a410 comp rx=0 tx=0).stop 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/2786541362 shutdown_connections 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/2786541362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 0x7f1b5810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/2786541362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b581087e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/2786541362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 0x7f1b58102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/2786541362 >> 172.21.15.16:0/2786541362 conn(0x7f1b58079760 msgr2=0x7f1b58079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/2786541362 shutdown_connections 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.688+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/2786541362 wait complete. 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.689+0000 7f1b5e8e0640 1 Processor -- start 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.689+0000 7f1b5e8e0640 1 -- start start 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.689+0000 7f1b5e8e0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 0x7f1b5819b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.689+0000 7f1b5e8e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.689+0000 7f1b5e8e0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 0x7f1b5819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b5e8e0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58112840 con 0x7f1b581083d0 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b5e8e0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581126c0 con 0x7f1b58108d20 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b5e8e0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581129c0 con 0x7f1b58102ae0 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39530/0 (socket says 172.21.15.16:39530) 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 -- 172.21.15.16:0/284726440 learned_addr learned my addr 172.21.15.16:0/284726440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b57fff640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 0x7f1b5819b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 0x7f1b5819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 -- 172.21.15.16:0/284726440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 msgr2=0x7f1b5819b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 0x7f1b5819b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 -- 172.21.15.16:0/284726440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 msgr2=0x7f1b5819bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 0x7f1b5819bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 -- 172.21.15.16:0/284726440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b581a64a0 con 0x7f1b581083d0 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b577fe640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005940 tx=0x7f1b4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.690+0000 7f1b557fa640 1 -- 172.21.15.16:0/284726440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c03f030 con 0x7f1b581083d0 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.691+0000 7f1b557fa640 1 -- 172.21.15.16:0/284726440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4c003c90 con 0x7f1b581083d0 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.691+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b581a6730 con 0x7f1b581083d0 2024-09-21T02:06:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.691+0000 7f1b557fa640 1 -- 172.21.15.16:0/284726440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c035070 con 0x7f1b581083d0 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.691+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b581a6c60 con 0x7f1b581083d0 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.693+0000 7f1b64e54740 1 Processor -- start 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.693+0000 7f1b64e54740 1 -- start start 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b557fa640 1 -- 172.21.15.16:0/284726440 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f1b4c042030 con 0x7f1b581083d0 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c0d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c0d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f08b84c10 con 0x564f08866f60 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f08b74040 con 0x564f0883e0a0 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b64e54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f08b77e50 con 0x564f08846280 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39546/0 (socket says 172.21.15.16:39546) 2024-09-21T02:06:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2289229275 learned_addr learned my addr 172.21.15.16:0/2289229275 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b57fff640 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c0d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2289229275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 msgr2=0x564f08c0d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c0d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2289229275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 msgr2=0x564f08c0d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c0d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2289229275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f088d2540 con 0x564f08866f60 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.694+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000ae90 tx=0x7f1b5000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caaa3da9b2dd3f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b3e7fc640 1 -- 172.21.15.16:0/2289229275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50003b70 con 0x564f08866f60 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b3e7fc640 1 -- 172.21.15.16:0/2289229275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b50003d10 con 0x564f08866f60 2024-09-21T02:06:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b3e7fc640 1 -- 172.21.15.16:0/2289229275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50010a80 con 0x564f08866f60 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b557fa640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 0x7f1b400b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b57fff640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 0x7f1b400b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b557fa640 1 -- 172.21.15.16:0/284726440 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f1b4c107c30 con 0x7f1b581083d0 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 -- 172.21.15.16:0/2289229275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 msgr2=0x564f08c0cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000ae90 tx=0x7f1b5000f790 comp rx=0 tx=0).stop 2024-09-21T02:06:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b57fff640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 0x7f1b400b8b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x564f08c0bb80 tx=0x7f1b4800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 -- 172.21.15.16:0/2289229275 shutdown_connections 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c0d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c0d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2289229275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c0cf00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.695+0000 7f1b64e54740 1 -- 172.21.15.16:0/2289229275 >> 172.21.15.16:0/2289229275 conn(0x564f0885b350 msgr2=0x564f08c0adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.696+0000 7f1b64e54740 1 -- 172.21.15.16:0/2289229275 shutdown_connections 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.696+0000 7f1b64e54740 1 -- 172.21.15.16:0/2289229275 wait complete. 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.696+0000 7f1b64e54740 1 -- start start 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.696+0000 7f1b64e54740 1 Processor -- start 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08866f60 0x564f08c24a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f089d55f0 con 0x564f08846280 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f08b0f010 con 0x564f0883e0a0 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f08c1b6e0 con 0x564f08866f60 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b64e54740 10 client.0 _ll_register_callbacks cb 0x564f08b6e340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39558/0 (socket says 172.21.15.16:39558) 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 -- 172.21.15.16:0/721990047 learned_addr learned my addr 172.21.15.16:0/721990047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:58.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b57fff640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08866f60 0x564f08c24a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 -- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08866f60 msgr2=0x564f08c24a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08866f60 0x564f08c24a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 -- 172.21.15.16:0/721990047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 msgr2=0x564f08c18a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 -- 172.21.15.16:0/721990047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f08b3ca00 con 0x564f08846280 2024-09-21T02:06:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b57fff640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b577fe640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c0c9e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cf369f78d9bd8d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b377fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.697+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c049040 con 0x564f08846280 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f1b4c003800 con 0x564f08846280 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b50003d10 con 0x564f08846280 2024-09-21T02:06:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b4c0cd7b0 con 0x564f08846280 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 10 client.15495 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f08c27630 con 0x564f08846280 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b50003d10 con 0x564f08846280 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f1b4c0c9330 con 0x564f08846280 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 10 client.15495 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 20 client.15495 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f088258e0 0x564f08c2b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 4 client.15495 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f08b86b60 con 0x564f088258e0 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.698+0000 7f1b57fff640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f088258e0 0x564f08c2b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.699+0000 7f1b57fff640 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f088258e0 0x564f08c2b080 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=361f8964486cdf5e server_cookie=ad648ba612890f08 in_seq=0 out_seq=0 2024-09-21T02:06:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.699+0000 7f1b377fe640 10 client.15495 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b377fe640 1 -- 172.21.15.16:0/721990047 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564f08b86b60 con 0x564f088258e0 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b377fe640 10 client.15495 handle_command_reply: tid=0 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 client.15495 shutdown 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 msgr2=0x564f08c18f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c0c9e80 comp rx=0 tx=0).stop 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 shutdown_connections 2024-09-21T02:06:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08866f60 0x564f08c24a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f088258e0 0x564f08c2b080 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 --2- 172.21.15.16:0/721990047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 >> 172.21.15.16:0/721990047 conn(0x564f0885b350 msgr2=0x564f08c100e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b377fe640 0 client.15495 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 shutdown_connections 2024-09-21T02:06:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 1 -- 172.21.15.16:0/721990047 wait complete. 2024-09-21T02:06:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.701+0000 7f1b64e54740 20 client.15495 trim_cache size 0 max 16384 2024-09-21T02:06:58.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 Processor -- start 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 -- start start 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f08c293c0 con 0x564f08866f60 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f08c29a90 con 0x564f0883e0a0 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b64e54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f08c2a160 con 0x564f08846280 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39564/0 (socket says 172.21.15.16:39564) 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2806794577 learned_addr learned my addr 172.21.15.16:0/2806794577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b57fff640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b577fe640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2806794577 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 msgr2=0x564f08c18f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2806794577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 msgr2=0x564f08c18a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b5ce7d640 1 -- 172.21.15.16:0/2806794577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f08c16d00 con 0x564f08866f60 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.719+0000 7f1b57fff640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.720+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000fc90 tx=0x7f1b5000a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320d7f7b60258e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.720+0000 7f1b3dffb640 1 -- 172.21.15.16:0/2806794577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5001d040 con 0x564f08866f60 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.720+0000 7f1b3dffb640 1 -- 172.21.15.16:0/2806794577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b50010be0 con 0x564f08866f60 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.720+0000 7f1b3dffb640 1 -- 172.21.15.16:0/2806794577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50018bc0 con 0x564f08866f60 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- 172.21.15.16:0/2806794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 msgr2=0x564f08c139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000fc90 tx=0x7f1b5000a540 comp rx=0 tx=0).stop 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- 172.21.15.16:0/2806794577 shutdown_connections 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f08866f60 0x564f08c139b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c18f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 --2- 172.21.15.16:0/2806794577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f0883e0a0 0x564f08c18a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- 172.21.15.16:0/2806794577 >> 172.21.15.16:0/2806794577 conn(0x564f0885b350 msgr2=0x564f08c28980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- 172.21.15.16:0/2806794577 shutdown_connections 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- 172.21.15.16:0/2806794577 wait complete. 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 -- start start 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.721+0000 7f1b64e54740 1 Processor -- start 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c2e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f08866f60 0x564f08c32060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f08c2af00 con 0x564f0883e0a0 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564f08c2a830 con 0x564f08866f60 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f08c2b5d0 con 0x564f08846280 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b64e54740 10 client.0 _ll_register_callbacks cb 0x7f1b4c03b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39566/0 (socket says 172.21.15.16:39566) 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 -- 172.21.15.16:0/286267682 learned_addr learned my addr 172.21.15.16:0/286267682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b5ce7d640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f08866f60 0x564f08c32060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b577fe640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c2e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 -- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 msgr2=0x564f08c2e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c2e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 -- 172.21.15.16:0/286267682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f08866f60 msgr2=0x564f08c32060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f08866f60 0x564f08c32060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b57fff640 1 -- 172.21.15.16:0/286267682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f08c326d0 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.722+0000 7f1b577fe640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c2e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b57fff640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806c6a0 tx=0x7f1b480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4878c72cc4840d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48035060 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f1b48017030 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b50010be0 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b4803c050 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 10 client.15501 fetch_fsmap learned FSMap version 27 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f08c33b20 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b50010be0 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f1b48030b80 con 0x564f0883e0a0 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 10 client.15501 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 10 client.15501 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.723+0000 7f1b64e54740 20 client.15501 populate_metadata read hostname 'smithi016' 2024-09-21T02:06:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f087fac90 0x564f08c996e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b64e54740 4 client.15501 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "status"}) -- 0x564f08b82610 con 0x564f087fac90 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b577fe640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f087fac90 0x564f08c996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b377fe640 10 client.15501 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b577fe640 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f087fac90 0x564f08c996e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44ef343c54e11aa6 server_cookie=50c86166e3c5363c in_seq=0 out_seq=0 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.724+0000 7f1b377fe640 1 -- 172.21.15.16:0/286267682 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564f08b82610 con 0x564f087fac90 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b377fe640 10 client.15501 handle_command_reply: tid=0 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 client.15501 shutdown 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 msgr2=0x564f08c2dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806c6a0 tx=0x7f1b480046c0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 shutdown_connections 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x564f087fac90 0x564f08c996e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564f08866f60 0x564f08c32060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f08846280 0x564f08c2e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 --2- 172.21.15.16:0/286267682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f0883e0a0 0x564f08c2dff0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b377fe640 0 client.15501 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:06:58.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 >> 172.21.15.16:0/286267682 conn(0x564f0885b350 msgr2=0x564f08c139b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 shutdown_connections 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 1 -- 172.21.15.16:0/286267682 wait complete. 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.725+0000 7f1b64e54740 20 client.15501 trim_cache size 0 max 16384 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24667, 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:06:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 100.91117324299999, 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 162.93614769800001 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:06:58.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 msgr2=0x7f1b400b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 0x7f1b400b8b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x564f08c0bb80 tx=0x7f1b4800b390 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 msgr2=0x7f1b5819b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005940 tx=0x7f1b4c0042f0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 shutdown_connections 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f1b400b6610 0x7f1b400b8b20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b58108d20 0x7f1b5819bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b581083d0 0x7f1b5819b750 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 --2- 172.21.15.16:0/284726440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b58102ae0 0x7f1b5819b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 >> 172.21.15.16:0/284726440 conn(0x7f1b58079760 msgr2=0x7f1b581044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 shutdown_connections 2024-09-21T02:06:58.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:06:58.726+0000 7f1b5e8e0640 1 -- 172.21.15.16:0/284726440 wait complete. 2024-09-21T02:06:59.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:06:59 smithi016 ceph-mon[31245]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-21T02:06:59.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:06:59 smithi161 ceph-mon[34061]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-21T02:06:59.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:06:59.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:06:59 smithi123 ceph-mon[33453]: pgmap v347: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-21T02:06:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 -- 172.21.15.16:0/760637564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9801094a0 msgr2=0x7ff9801098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 --2- 172.21.15.16:0/760637564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9801094a0 0x7ff9801098b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff970009900 tx=0x7ff97002a410 comp rx=0 tx=0).stop 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 -- 172.21.15.16:0/760637564 shutdown_connections 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 --2- 172.21.15.16:0/760637564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff980109df0 0x7ff980110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 --2- 172.21.15.16:0/760637564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9801094a0 0x7ff9801098b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 --2- 172.21.15.16:0/760637564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 0x7ff980103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 -- 172.21.15.16:0/760637564 >> 172.21.15.16:0/760637564 conn(0x7ff98007b850 msgr2=0x7ff98007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 -- 172.21.15.16:0/760637564 shutdown_connections 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.250+0000 7ff988045640 1 -- 172.21.15.16:0/760637564 wait complete. 2024-09-21T02:07:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 Processor -- start 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 -- start start 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 0x7ff98019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9801094a0 0x7ff98019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff980113960 con 0x7ff980109df0 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801137e0 con 0x7ff9801094a0 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff988045640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff980113ae0 con 0x7ff980103b60 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff9865e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff9865e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39578/0 (socket says 172.21.15.16:39578) 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff9865e2640 1 -- 172.21.15.16:0/233333905 learned_addr learned my addr 172.21.15.16:0/233333905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff985de1640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 0x7ff98019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.251+0000 7ff9855e0640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9801094a0 0x7ff98019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 -- 172.21.15.16:0/233333905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 msgr2=0x7ff98019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 0x7ff98019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 -- 172.21.15.16:0/233333905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9801094a0 msgr2=0x7ff98019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9801094a0 0x7ff98019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 -- 172.21.15.16:0/233333905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9801a02a0 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff9865e2640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff97800cce0 tx=0x7ff97800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff976ffd640 1 -- 172.21.15.16:0/233333905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97800f450 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff976ffd640 1 -- 172.21.15.16:0/233333905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff97800fa80 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9801a0590 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff976ffd640 1 -- 172.21.15.16:0/233333905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978018ad0 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.252+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801a6db0 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.255+0000 7ff98e5b9740 1 Processor -- start 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff976ffd640 1 -- 172.21.15.16:0/233333905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7ff97800f5f0 con 0x7ff980109df0 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 -- start start 2024-09-21T02:07:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac41ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac420440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac420980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caac397c10 con 0x55caac059280 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55caac387040 con 0x55caac079f60 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55caac38ae50 con 0x55caac0510a0 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac420980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac420980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39594/0 (socket says 172.21.15.16:39594) 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff9855e0640 1 -- 172.21.15.16:0/748892576 learned_addr learned my addr 172.21.15.16:0/748892576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff985de1640 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac420440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.256+0000 7ff9855e0640 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac41ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff976ffd640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 0x7ff96c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 -- 172.21.15.16:0/748892576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 msgr2=0x55caac41ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac41ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 -- 172.21.15.16:0/748892576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 msgr2=0x55caac420980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac420980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 -- 172.21.15.16:0/748892576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caac0e5540 con 0x55caac0510a0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff9865e2640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 0x7ff96c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff985de1640 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac420440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009c20 tx=0x7ff97c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f26734ab61ccc6db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.257+0000 7ff976ffd640 1 -- 172.21.15.16:0/233333905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff9780e7e80 con 0x7ff980109df0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.258+0000 7ff9865e2640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 0x7ff96c0b8b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff9780adb10 tx=0x7ff9780a23e0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.258+0000 7ff96bfff640 1 -- 172.21.15.16:0/748892576 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c004050 con 0x55caac0510a0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.258+0000 7ff96bfff640 1 -- 172.21.15.16:0/748892576 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff97c005020 con 0x55caac0510a0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.258+0000 7ff96bfff640 1 -- 172.21.15.16:0/748892576 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c004bc0 con 0x55caac0510a0 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- 172.21.15.16:0/748892576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 msgr2=0x55caac420440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac420440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009c20 tx=0x7ff97c011810 comp rx=0 tx=0).stop 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- 172.21.15.16:0/748892576 shutdown_connections 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac420980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac420440 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/748892576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac41ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- 172.21.15.16:0/748892576 >> 172.21.15.16:0/748892576 conn(0x55caac06e350 msgr2=0x55caac41ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- 172.21.15.16:0/748892576 shutdown_connections 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- 172.21.15.16:0/748892576 wait complete. 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 -- start start 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.259+0000 7ff98e5b9740 1 Processor -- start 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac42b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac059280 0x55caac42bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caac1e85f0 con 0x55caac079f60 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55caac322010 con 0x55caac059280 2024-09-21T02:07:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55caac42e6e0 con 0x55caac0510a0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff98e5b9740 10 client.0 _ll_register_callbacks cb 0x55caac381340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39602/0 (socket says 172.21.15.16:39602) 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 -- 172.21.15.16:0/1087336705 learned_addr learned my addr 172.21.15.16:0/1087336705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff985de1640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac059280 0x55caac42bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac42b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 -- 172.21.15.16:0/1087336705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 msgr2=0x55caac42b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac42b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 -- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac059280 msgr2=0x55caac42bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac059280 0x55caac42bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.260+0000 7ff9855e0640 1 -- 172.21.15.16:0/1087336705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caac34fa00 con 0x55caac079f60 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9855e0640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff97002a930 tx=0x7ff97002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a9fe4738427b5a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970035040 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff970004480 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff97c005020 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff9700035f0 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff98e5b9740 10 client.15513 fetch_fsmap learned FSMap version 27 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55caac43a620 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff97c005020 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7ff97003d570 con 0x55caac079f60 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.261+0000 7ff98e5b9740 10 client.15513 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff98e5b9740 10 client.15513 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff98e5b9740 20 client.15513 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac0388e0 0x55caac43e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff98e5b9740 4 client.15513 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caac399b60 con 0x55caac0388e0 2024-09-21T02:07:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff985de1640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac0388e0 0x55caac43e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff9697fa640 10 client.15513 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.262+0000 7ff985de1640 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac0388e0 0x55caac43e030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30d42c9ab028a98b server_cookie=69dff3bbb4771b8b in_seq=0 out_seq=0 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.263+0000 7ff9697fa640 1 -- 172.21.15.16:0/1087336705 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55caac399b60 con 0x55caac0388e0 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.263+0000 7ff9697fa640 10 client.15513 handle_command_reply: tid=0 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.263+0000 7ff98e5b9740 1 client.15513 shutdown 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.263+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 msgr2=0x55caac4379d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.263+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff97002a930 tx=0x7ff97002a410 comp rx=0 tx=0).stop 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 shutdown_connections 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac0388e0 0x55caac43e030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac079f60 0x55caac4379d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac059280 0x55caac42bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/1087336705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac0510a0 0x55caac42b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff9697fa640 0 client.15513 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 >> 172.21.15.16:0/1087336705 conn(0x55caac06e350 msgr2=0x55caac4230e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 shutdown_connections 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 1 -- 172.21.15.16:0/1087336705 wait complete. 2024-09-21T02:07:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.264+0000 7ff98e5b9740 20 client.15513 trim_cache size 0 max 16384 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 Processor -- start 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 -- start start 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac0510a0 0x55caac42acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac079f60 0x55caac437e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caac43c370 con 0x55caac059280 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55caac43ca40 con 0x55caac0510a0 2024-09-21T02:07:00.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.282+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55caac43d110 con 0x55caac079f60 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39604/0 (socket says 172.21.15.16:39604) 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 -- 172.21.15.16:0/564997342 learned_addr learned my addr 172.21.15.16:0/564997342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff9855e0640 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac079f60 0x55caac437e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 -- 172.21.15.16:0/564997342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac079f60 msgr2=0x55caac437e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff9855e0640 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac0510a0 0x55caac42acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac079f60 0x55caac437e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 -- 172.21.15.16:0/564997342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac0510a0 msgr2=0x55caac42acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac0510a0 0x55caac42acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 -- 172.21.15.16:0/564997342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caac4221f0 con 0x55caac059280 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff985de1640 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009250 tx=0x7ff97c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec154b9ea3ddbb9d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff96b7fe640 1 -- 172.21.15.16:0/564997342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c032070 con 0x55caac059280 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff96b7fe640 1 -- 172.21.15.16:0/564997342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff97c005610 con 0x55caac059280 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.283+0000 7ff96b7fe640 1 -- 172.21.15.16:0/564997342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c004e30 con 0x55caac059280 2024-09-21T02:07:00.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 -- 172.21.15.16:0/564997342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 msgr2=0x55caac437940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009250 tx=0x7ff97c009930 comp rx=0 tx=0).stop 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 -- 172.21.15.16:0/564997342 shutdown_connections 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac079f60 0x55caac437e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac059280 0x55caac437940 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/564997342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac0510a0 0x55caac42acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 -- 172.21.15.16:0/564997342 >> 172.21.15.16:0/564997342 conn(0x55caac06e350 msgr2=0x55caac42b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 -- 172.21.15.16:0/564997342 shutdown_connections 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.284+0000 7ff98e5b9740 1 -- 172.21.15.16:0/564997342 wait complete. 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 -- start start 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 Processor -- start 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 0x55caac43ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac440480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55caac43deb0 con 0x55caac0510a0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55caac43d7e0 con 0x55caac079f60 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55caac43e580 con 0x55caac059280 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff98e5b9740 10 client.0 _ll_register_callbacks cb 0x7ff9700317e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff9855e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39620/0 (socket says 172.21.15.16:39620) 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.285+0000 7ff9855e0640 1 -- 172.21.15.16:0/109629060 learned_addr learned my addr 172.21.15.16:0/109629060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff985de1640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 0x55caac43ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 -- 172.21.15.16:0/109629060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 msgr2=0x55caac43ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 0x55caac43ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 -- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 msgr2=0x55caac440480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac440480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 -- 172.21.15.16:0/109629060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caac440bf0 con 0x55caac0510a0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff985de1640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 0x55caac43ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9855e0640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff97003dae0 tx=0x7ff9700316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddd47055d50c590 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9697fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:00.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970035040 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff97003d660 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff97c005610 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff970031a20 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff98e5b9740 10 client.15519 fetch_fsmap learned FSMap version 27 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55caac4a9500 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.286+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff97c005610 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7ff970031c50 con 0x55caac0510a0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 10 client.15519 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 20 client.15519 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac00dc90 0x55caac4acb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 4 client.15519 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "status"}) -- 0x55caac395610 con 0x55caac00dc90 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff985de1640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac00dc90 0x55caac4acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:00.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff9697fa640 10 client.15519 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.287+0000 7ff985de1640 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac00dc90 0x55caac4acb30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f17e0bf7ce932789 server_cookie=436b2801d273a0af in_seq=0 out_seq=0 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff9697fa640 1 -- 172.21.15.16:0/109629060 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55caac395610 con 0x55caac00dc90 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff9697fa640 10 client.15519 handle_command_reply: tid=0 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 client.15519 shutdown 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 msgr2=0x55caac446a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff97003dae0 tx=0x7ff9700316f0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 shutdown_connections 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x55caac00dc90 0x55caac4acb30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55caac079f60 0x55caac440480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caac059280 0x55caac43ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff9697fa640 0 client.15519 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 --2- 172.21.15.16:0/109629060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55caac0510a0 0x55caac446a90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 >> 172.21.15.16:0/109629060 conn(0x55caac06e350 msgr2=0x55caac4269c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 shutdown_connections 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 1 -- 172.21.15.16:0/109629060 wait complete. 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.288+0000 7ff98e5b9740 20 client.15519 trim_cache size 0 max 16384 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24683, 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:00.295 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 103.48143006700001, 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 167.081234736 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 msgr2=0x7ff96c0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 0x7ff96c0b8b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff9780adb10 tx=0x7ff9780a23e0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 msgr2=0x7ff98019fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff97800cce0 tx=0x7ff97800a620 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 shutdown_connections 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff96c0b6610 0x7ff96c0b8b20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff980109df0 0x7ff98019fc30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9801094a0 0x7ff98019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 --2- 172.21.15.16:0/233333905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff980103b60 0x7ff98019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 >> 172.21.15.16:0/233333905 conn(0x7ff98007b850 msgr2=0x7ff980105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 shutdown_connections 2024-09-21T02:07:00.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:00.289+0000 7ff988045640 1 -- 172.21.15.16:0/233333905 wait complete. 2024-09-21T02:07:01.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:07:01.384 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:01.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:01 smithi161 ceph-mon[34061]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:01.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:01 smithi016 ceph-mon[31245]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:01.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:01 smithi123 ceph-mon[33453]: pgmap v348: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.850+0000 7f84511cf640 1 -- 172.21.15.16:0/1435344460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 msgr2=0x7f844c10d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.850+0000 7f84511cf640 1 --2- 172.21.15.16:0/1435344460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c10d8d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8440009900 tx=0x7f844002a410 comp rx=0 tx=0).stop 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 -- 172.21.15.16:0/1435344460 shutdown_connections 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 --2- 172.21.15.16:0/1435344460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 0x7f844c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 --2- 172.21.15.16:0/1435344460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c10d8d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 --2- 172.21.15.16:0/1435344460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 -- 172.21.15.16:0/1435344460 >> 172.21.15.16:0/1435344460 conn(0x7f844c079760 msgr2=0x7f844c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 -- 172.21.15.16:0/1435344460 shutdown_connections 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 -- 172.21.15.16:0/1435344460 wait complete. 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.851+0000 7f84511cf640 1 Processor -- start 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 -- start start 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 0x7f844c1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f844c113950 con 0x7f844c105380 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f844c1137d0 con 0x7f844c104a30 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f84511cf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f844c113ad0 con 0x7f844c10de60 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39634/0 (socket says 172.21.15.16:39634) 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f844ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f844a59c640 1 -- 172.21.15.16:0/818469102 learned_addr learned my addr 172.21.15.16:0/818469102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.852+0000 7f844b59e640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 0x7f844c1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 -- 172.21.15.16:0/818469102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 msgr2=0x7f844c1031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 0x7f844c1031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 -- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 msgr2=0x7f844c102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 -- 172.21.15.16:0/818469102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844c1a8680 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844a59c640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f844ad9d640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c102770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d6a0 tx=0x7f843c00db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f8433fff640 1 -- 172.21.15.16:0/818469102 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c017070 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844c1a8970 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844c1a8ed0 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f8433fff640 1 -- 172.21.15.16:0/818469102 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f843c009cb0 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.853+0000 7f8433fff640 1 -- 172.21.15.16:0/818469102 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c020060 con 0x7f844c104a30 2024-09-21T02:07:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 Processor -- start 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 -- start start 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbbbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbbc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56428db33c10 con 0x56428d7f5280 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56428db23040 con 0x56428d815f60 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8457743740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56428db26e50 con 0x56428d7ed0a0 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f8433fff640 1 -- 172.21.15.16:0/818469102 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f843c009820 con 0x7f844c104a30 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39648/0 (socket says 172.21.15.16:39648) 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 -- 172.21.15.16:0/3170629907 learned_addr learned my addr 172.21.15.16:0/3170629907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844b59e640 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbbbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 -- 172.21.15.16:0/3170629907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 msgr2=0x56428dbbc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbbc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 -- 172.21.15.16:0/3170629907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 msgr2=0x56428dbbbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbbbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.856+0000 7f844a59c640 1 -- 172.21.15.16:0/3170629907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56428d881540 con 0x56428d7f5280 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f844a59c640 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8440009230 tx=0x7f8440031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a85e2f48dde32d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f84317fa640 1 -- 172.21.15.16:0/3170629907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440035070 con 0x56428d7f5280 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f84317fa640 1 -- 172.21.15.16:0/3170629907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f84400044c0 con 0x56428d7f5280 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f84317fa640 1 -- 172.21.15.16:0/3170629907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440003e10 con 0x56428d7f5280 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f8433fff640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 0x7f84340b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.857+0000 7f844b59e640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 0x7f84340b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8433fff640 1 -- 172.21.15.16:0/818469102 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f843c0e7140 con 0x7f844c104a30 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- 172.21.15.16:0/3170629907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 msgr2=0x56428dbbc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f844b59e640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 0x7f84340b8a70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8444005fd0 tx=0x7f8444009600 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8440009230 tx=0x7f8440031400 comp rx=0 tx=0).stop 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- 172.21.15.16:0/3170629907 shutdown_connections 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbbc980 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbbc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 --2- 172.21.15.16:0/3170629907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbbbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- 172.21.15.16:0/3170629907 >> 172.21.15.16:0/3170629907 conn(0x56428d80a350 msgr2=0x56428dbb9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- 172.21.15.16:0/3170629907 shutdown_connections 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- 172.21.15.16:0/3170629907 wait complete. 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.858+0000 7f8457743740 1 -- start start 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 Processor -- start 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbd3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56428d9845f0 con 0x56428d7f5280 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56428dabe010 con 0x56428d815f60 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56428dbca680 con 0x56428d7ed0a0 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f8457743740 10 client.0 _ll_register_callbacks cb 0x56428db1d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39660/0 (socket says 172.21.15.16:39660) 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 -- 172.21.15.16:0/1946288714 learned_addr learned my addr 172.21.15.16:0/1946288714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844b59e640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbd3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844ad9d640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 -- 172.21.15.16:0/1946288714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 msgr2=0x56428dbc7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 -- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 msgr2=0x56428dbd3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbd3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.859+0000 7f844a59c640 1 -- 172.21.15.16:0/1946288714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56428daeba00 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f844a59c640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f844002a450 tx=0x7f8440009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40a62071b2d5503 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440035040 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56428db35b60 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f8440003b90 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8440003d90 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 10 client.15531 fetch_fsmap learned FSMap version 27 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56428dbd6640 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56428db35b60 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f8440045ad0 con 0x56428d7f5280 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 10 client.15531 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 10 client.15531 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:07:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.860+0000 7f8457743740 20 client.15531 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7d48e0 0x56428dbd9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f8457743740 4 client.15531 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56428db31610 con 0x56428d7d48e0 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f844ad9d640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7d48e0 0x56428dbd9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f842eff5640 10 client.15531 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.861+0000 7f844ad9d640 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7d48e0 0x56428dbd9b30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7a7108373ed688 server_cookie=8f93472dde017c1d in_seq=0 out_seq=0 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.862+0000 7f842eff5640 1 -- 172.21.15.16:0/1946288714 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56428db31610 con 0x56428d7d48e0 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.862+0000 7f842eff5640 10 client.15531 handle_command_reply: tid=0 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.862+0000 7f8457743740 1 client.15531 shutdown 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 msgr2=0x56428dbc7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f844002a450 tx=0x7f8440009680 comp rx=0 tx=0).stop 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 shutdown_connections 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7d48e0 0x56428dbd9b30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbd3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7f5280 0x56428dbc7fa0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f842eff5640 0 client.15531 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 --2- 172.21.15.16:0/1946288714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 >> 172.21.15.16:0/1946288714 conn(0x56428d80a350 msgr2=0x56428dbbf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 shutdown_connections 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 1 -- 172.21.15.16:0/1946288714 wait complete. 2024-09-21T02:07:01.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.863+0000 7f8457743740 20 client.15531 trim_cache size 0 max 16384 2024-09-21T02:07:01.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 Processor -- start 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 -- start start 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d7f5280 0x56428dbc7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d815f60 0x56428dbd38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56428dbd7e70 con 0x56428d7ed0a0 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56428dbd8540 con 0x56428d7f5280 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f8457743740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56428dbd8c10 con 0x56428d815f60 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f844ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f844ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39672/0 (socket says 172.21.15.16:39672) 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.881+0000 7f844ad9d640 1 -- 172.21.15.16:0/2728321401 learned_addr learned my addr 172.21.15.16:0/2728321401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844a59c640 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d7f5280 0x56428dbc7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844b59e640 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d815f60 0x56428dbd38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 -- 172.21.15.16:0/2728321401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d815f60 msgr2=0x56428dbd38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d815f60 0x56428dbd38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 -- 172.21.15.16:0/2728321401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d7f5280 msgr2=0x56428dbc7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d7f5280 0x56428dbc7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 -- 172.21.15.16:0/2728321401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56428dbc8b20 con 0x56428d7ed0a0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f843c0acb90 tx=0x7f843c0acc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ecf2ca4ebddd34 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f8430ff9640 1 -- 172.21.15.16:0/2728321401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c0b18c0 con 0x56428d7ed0a0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f8430ff9640 1 -- 172.21.15.16:0/2728321401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f843c0b6800 con 0x56428d7ed0a0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.882+0000 7f8430ff9640 1 -- 172.21.15.16:0/2728321401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c0ac0f0 con 0x56428d7ed0a0 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- 172.21.15.16:0/2728321401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 msgr2=0x56428dbc7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f843c0acb90 tx=0x7f843c0acc60 comp rx=0 tx=0).stop 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- 172.21.15.16:0/2728321401 shutdown_connections 2024-09-21T02:07:01.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d815f60 0x56428dbd38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d7f5280 0x56428dbc7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 --2- 172.21.15.16:0/2728321401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbc7a40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- 172.21.15.16:0/2728321401 >> 172.21.15.16:0/2728321401 conn(0x56428d80a350 msgr2=0x56428dbc1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- 172.21.15.16:0/2728321401 shutdown_connections 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- 172.21.15.16:0/2728321401 wait complete. 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.883+0000 7f8457743740 1 -- start start 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 Processor -- start 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 0x56428dbdb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbdbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56428dbd99b0 con 0x56428d7ed0a0 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56428dbd92e0 con 0x56428d815f60 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56428dbda080 con 0x56428d7f5280 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f8457743740 10 client.0 _ll_register_callbacks cb 0x7f8440032ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:01.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39680/0 (socket says 172.21.15.16:39680) 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 -- 172.21.15.16:0/2120411364 learned_addr learned my addr 172.21.15.16:0/2120411364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844a59c640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 0x56428dbdb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844b59e640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbdbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 -- 172.21.15.16:0/2120411364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 msgr2=0x56428dbdb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 0x56428dbdb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 -- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 msgr2=0x56428dbdbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbdbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844ad9d640 1 -- 172.21.15.16:0/2120411364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56428dbdc250 con 0x56428d7ed0a0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.884+0000 7f844a59c640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 0x56428dbdb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f844ad9d640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f843c005cc0 tx=0x7f843c009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2479d08ae44cbc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:01.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c0042b0 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f843c05c1b0 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f843c0b6800 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f843c0a9100 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 10 client.15537 fetch_fsmap learned FSMap version 27 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56428dbe1b60 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f843c0b6800 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) ==== 8370+0+0 (secure 0 0 0) 0x7f843c0d41b0 con 0x56428d7ed0a0 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 10 client.15537 fetch_fsmap finished waiting for FSMap version 27 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 10 client.15537 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 20 client.15537 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7a9c90 0x56428dc48000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 4 client.15537 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:01.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.885+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "status"}) -- 0x56428db2ea40 con 0x56428d7a9c90 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f844a59c640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7a9c90 0x56428dc48000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f844a59c640 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7a9c90 0x56428dc48000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc6e18e36e92fe72 server_cookie=3a60bad05f92c67 in_seq=0 out_seq=0 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f842eff5640 10 client.15537 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f842eff5640 1 -- 172.21.15.16:0/2120411364 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56428db2ea40 con 0x56428d7a9c90 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f842eff5640 10 client.15537 handle_command_reply: tid=0 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f8457743740 1 client.15537 shutdown 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 msgr2=0x56428dbe22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f843c005cc0 tx=0x7f843c009f40 comp rx=0 tx=0).stop 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.886+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 shutdown_connections 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56428d7a9c90 0x56428dc48000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56428d815f60 0x56428dbdbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56428d7f5280 0x56428dbdb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 --2- 172.21.15.16:0/2120411364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56428d7ed0a0 0x56428dbe22d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f842eff5640 0 client.15537 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 >> 172.21.15.16:0/2120411364 conn(0x56428d80a350 msgr2=0x56428dbb9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 shutdown_connections 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 1 -- 172.21.15.16:0/2120411364 wait complete. 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f8457743740 20 client.15537 trim_cache size 0 max 16384 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24710, 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 147.238955796, 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 84, 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 156.434328905 2024-09-21T02:07:01.897 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 msgr2=0x7f84340b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 0x7f84340b8a70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8444005fd0 tx=0x7f8444009600 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 msgr2=0x7f844c102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c102770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d6a0 tx=0x7f843c00db60 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 shutdown_connections 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f84340b6560 0x7f84340b8a70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c10de60 0x7f844c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844c105380 0x7f844c102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 --2- 172.21.15.16:0/818469102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f844c104a30 0x7f844c102770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.887+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 >> 172.21.15.16:0/818469102 conn(0x7f844c079760 msgr2=0x7f844c1009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.888+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 shutdown_connections 2024-09-21T02:07:01.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:01.888+0000 7f84511cf640 1 -- 172.21.15.16:0/818469102 wait complete. 2024-09-21T02:07:02.702 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-21T02:07:02.714 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-21T02:07:02.714 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-09-21T02:07:02.714 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-09-21T02:07:02.714 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T02:07:02.714 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:02.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:02.714 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-21T02:07:02.733 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-21T02:07:02.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:02.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-21T02:07:02.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:02.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-09-21T02:07:02.911 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-21T02:07:02.911 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-21T02:07:02.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T02:07:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:03.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:03 smithi016 ceph-mon[31245]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T02:07:03.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:03 smithi123 ceph-mon[33453]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 -- 172.21.15.16:0/2982951949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594109e00 msgr2=0x7f1594110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2982951949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594109e00 0x7f1594110630 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f158c009ca0 tx=0x7f158c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 -- 172.21.15.16:0/2982951949 shutdown_connections 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2982951949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594109e00 0x7f1594110630 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2982951949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15941094b0 0x7f15941098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2982951949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1594103bc0 0x7f1594103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 -- 172.21.15.16:0/2982951949 >> 172.21.15.16:0/2982951949 conn(0x7f159407b910 msgr2=0x7f159407bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 -- 172.21.15.16:0/2982951949 shutdown_connections 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.655+0000 7f159a8c5640 1 -- 172.21.15.16:0/2982951949 wait complete. 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.656+0000 7f159a8c5640 1 Processor -- start 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.656+0000 7f159a8c5640 1 -- start start 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15941094b0 0x7f1594075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 0x7f1594076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15941139c0 con 0x7f1594103bc0 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1594113840 con 0x7f1594109e00 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1594113b40 con 0x7f15941094b0 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39698/0 (socket says 172.21.15.16:39698) 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 -- 172.21.15.16:0/2587264203 learned_addr learned my addr 172.21.15.16:0/2587264203 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f15937fe640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15941094b0 0x7f1594075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1598e62640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 0x7f1594076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 -- 172.21.15.16:0/2587264203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15941094b0 msgr2=0x7f1594075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15941094b0 0x7f1594075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 -- 172.21.15.16:0/2587264203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 msgr2=0x7f1594076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 0x7f1594076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 -- 172.21.15.16:0/2587264203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15941a86d0 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1598e62640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 0x7f1594076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f1593fff640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f158400d770 tx=0x7f158400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584017070 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.657+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15941a89c0 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.658+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15840054b0 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.658+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584004c60 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.658+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15941a8f00 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.659+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1594103fd0 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.660+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f1584004e00 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.661+0000 7f15917fa640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 0x7f157c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.661+0000 7f15937fe640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 0x7f157c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.662+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f15840e7680 con 0x7f1594103bc0 2024-09-21T02:07:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.662+0000 7f15937fe640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 0x7f157c0b8a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1594109290 tx=0x7f158800a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:03.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:03 smithi161 ceph-mon[34061]: pgmap v349: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-21T02:07:03.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.664+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f15840ada20 con 0x7f1594103bc0 2024-09-21T02:07:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.896+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1594076a10 con 0x7f1594103bc0 2024-09-21T02:07:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.896+0000 7f15917fa640 1 -- 172.21.15.16:0/2587264203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f15840b1e00 con 0x7f1594103bc0 2024-09-21T02:07:03.897 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 msgr2=0x7f157c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 0x7f157c0b8a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1594109290 tx=0x7f158800a2d0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 msgr2=0x7f1594075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f158400d770 tx=0x7f158400dc30 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 shutdown_connections 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f157c0b6560 0x7f157c0b8a70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1594109e00 0x7f1594076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15941094b0 0x7f1594075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 --2- 172.21.15.16:0/2587264203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1594103bc0 0x7f1594075710 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 >> 172.21.15.16:0/2587264203 conn(0x7f159407b910 msgr2=0x7f15941075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 shutdown_connections 2024-09-21T02:07:03.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:03.899+0000 7f159a8c5640 1 -- 172.21.15.16:0/2587264203 wait complete. 2024-09-21T02:07:04.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:04 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2587264203' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:07:04.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:04 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2587264203' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T02:07:04.654 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-21T02:07:04.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:04.655 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-09-21T02:07:04.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:04 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2587264203' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T02:07:04.673 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T02:07:04.673 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-09-21T02:07:04.674 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42102sec preferred_lft 42102sec 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-09-21T02:07:04.675 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-09-21T02:07:04.675 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T02:07:04.675 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:07:04.675 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:07:04.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-09-21T02:07:04.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-09-21T02:07:04.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-09-21T02:07:04.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T02:07:04.676 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:07:04.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:07:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:04.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:04.825 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T02:07:04.849 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-09-21T02:07:04.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:04.851 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-09-21T02:07:04.908 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-09-21T02:07:04.909 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T02:07:04.909 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T02:07:04.909 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T02:07:04.909 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T02:07:04.909 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:07:04.909 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:07:04.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T02:07:04.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T02:07:04.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T02:07:04.910 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:07:04.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:07:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:05.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:05.038 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-21T02:07:05.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:05.054 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-09-21T02:07:05.110 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:07:05.110 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:07:05.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:05.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-21T02:07:05.110 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:07:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:07:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:05.229 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-21T02:07:05.229 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:07:05.229 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-21T02:07:05.230 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:07:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:07:05.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:05 smithi016 ceph-mon[31245]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:05.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:05.348 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T02:07:05.348 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-09-21T02:07:05.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-09-21T02:07:05.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T02:07:05.348 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-09-21T02:07:05.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T02:07:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:05.427 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-09-21T02:07:05.427 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:07:05.427 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:05.443 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-21T02:07:05.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T02:07:05.443 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:05.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:07:05.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:05 smithi123 ceph-mon[33453]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:05.565 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:07:05.566 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:05 smithi161 ceph-mon[34061]: pgmap v350: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:07:05.622 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:07:05.623 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:07:05.624 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:05.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:05.626 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:07:05.681 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T02:07:05.682 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/fcbd249e-f272-4381-beda-18705a85f32c --client_fs=cephfs) 2024-09-21T02:07:05.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:07:05.749 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:07:05.807 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:07:05.807 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:07:05.808 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:07:05.809 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:05.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:05.810 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:07:05.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-09-21T02:07:05.818+0000 7f80e1d82480 -1 init, newargv = 0x7f80c4056f60 newargc=15 2024-09-21T02:07:05.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[87558]: starting ceph client 2024-09-21T02:07:05.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[87558]: starting fuse 2024-09-21T02:07:06.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-09-21T02:07:06.968 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T02:07:07.029 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-09-21T02:07:07.029 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-09-21T02:07:07.029 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-09-21T02:07:07.029 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-09-21T02:07:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/fe4e371bf21ab734cbedcd99b412d27580a93688cc8f44541565ad4860f56ee6/merged 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/2052e93c585489737988bfa5607ca152102cc41c10945b80086a1d298c5e6790/merged 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T02:07:07.031 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/05ea5618fedc089bd49f8b2a46edf11a644c8432a9a8bf3138a0fa04e666cc95/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/937b0e634c254e620ffdf4a94a16a42ae08b911c246ca3304b670ddb1d82d949/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c76c01c56a969ef65bbfa06e31533ed5a6951758c0ffb9f332cf691d0d2a1311/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/161dd3ef1ffbeeaa9e3e33d0919e3bb2d1580e764c14588ce59460ba13821aa7/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/649f7dacc350379cb0ca3c0024d84f08073db7626f4832a56abd44e259be9ac3/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/ac2eeb1dcdfd73228e21767afb39c7cf41ba303f0df6c39a03e16770b565b0f2/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b06ff9e49c2ade3c6352b2ba398119e8b3987babd998c48a9da18e6747454d93/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/33af3a4202be1f5d6f5d74ba3709062b3dd3ac12d72574f4792a1b950c13674c/merged 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:07.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:07.033 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-09-21T02:07:07.088 INFO:teuthology.orchestra.run.smithi016.stdout:108 2024-09-21T02:07:07.088 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [108] 2024-09-21T02:07:07.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-09-21T02:07:07.088 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-09-21T02:07:07.088 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T02:07:07.089 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-09-21T02:07:07.090 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-09-21T02:07:07.221 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.87558.asok 2024-09-21T02:07:07.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T02:07:07.232 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:07 smithi016 ceph-mon[31245]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:07.233 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.87558.asok 2024-09-21T02:07:07.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:07.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.87558.asok status 2024-09-21T02:07:07.382 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:07.382 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "87558", 2024-09-21T02:07:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/volumes/qa/sv_0/fcbd249e-f272-4381-beda-18705a85f32c" 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 15549, 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-09-21T02:07:07.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 15549 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:07:07.385 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2916906223 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-09-21T02:07:07.386 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2916906223 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.15549 192.168.0.1:0/2916906223", 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2916906223", 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 5, 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 27, 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 86, 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-09-21T02:07:07.387 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-09-21T02:07:07.388 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-09-21T02:07:07.388 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:07.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:07.391 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-09-21T02:07:07.391 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:07.408 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-09-21T02:07:07.409 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:07.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:07:07.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:07.477 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T02:07:07.496 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T02:07:07.497 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:07:07.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-09-21T02:07:07.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:07 smithi123 ceph-mon[33453]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:07.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 -- 172.21.15.16:0/4087698298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01029b0 msgr2=0x7fa0f0102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:07.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 --2- 172.21.15.16:0/4087698298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01029b0 0x7fa0f0102dc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009900 tx=0x7fa0ec02a410 comp rx=0 tx=0).stop 2024-09-21T02:07:07.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 -- 172.21.15.16:0/4087698298 shutdown_connections 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 --2- 172.21.15.16:0/4087698298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f0108c30 0x7fa0f010f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 --2- 172.21.15.16:0/4087698298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f01082e0 0x7fa0f01086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 --2- 172.21.15.16:0/4087698298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01029b0 0x7fa0f0102dc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 -- 172.21.15.16:0/4087698298 >> 172.21.15.16:0/4087698298 conn(0x7fa0f007a740 msgr2=0x7fa0f007ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 -- 172.21.15.16:0/4087698298 shutdown_connections 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.634+0000 7fa0f732b640 1 -- 172.21.15.16:0/4087698298 wait complete. 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 Processor -- start 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 -- start start 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01029b0 0x7fa0f019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:07.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 0x7fa0f019bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f01127a0 con 0x7fa0f01082e0 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f0112620 con 0x7fa0f01029b0 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f732b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f0112920 con 0x7fa0f0108c30 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f48c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f48c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57604/0 (socket says 172.21.15.16:57604) 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.635+0000 7fa0f48c6640 1 -- 172.21.15.16:0/2592317056 learned_addr learned my addr 172.21.15.16:0/2592317056 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f58c8640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 0x7fa0f019bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f50c7640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01029b0 0x7fa0f019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 -- 172.21.15.16:0/2592317056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 msgr2=0x7fa0f019bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:07.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 0x7fa0f019bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 -- 172.21.15.16:0/2592317056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01029b0 msgr2=0x7fa0f019b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01029b0 0x7fa0f019b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 -- 172.21.15.16:0/2592317056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f01a63d0 con 0x7fa0f01082e0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f58c8640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 0x7fa0f019bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f48c6640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c8c0 tx=0x7fa0e000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007bb0 con 0x7fa0f01082e0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0e0007d50 con 0x7fa0f01082e0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01a66c0 con 0x7fa0f01082e0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00186a0 con 0x7fa0f01082e0 2024-09-21T02:07:07.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.636+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f01a6c60 con 0x7fa0f01082e0 2024-09-21T02:07:07.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.637+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f0102dc0 con 0x7fa0f01082e0 2024-09-21T02:07:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.639+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7fa0e000fa00 con 0x7fa0f01082e0 2024-09-21T02:07:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.639+0000 7fa0e67fc640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 0x7fa0dc0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.640+0000 7fa0f50c7640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 0x7fa0dc0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.640+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fa0e00e7ba0 con 0x7fa0f01082e0 2024-09-21T02:07:07.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.640+0000 7fa0f50c7640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 0x7fa0dc0b8af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec006010 tx=0x7fa0ec009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:07.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.643+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa0e0014030 con 0x7fa0f01082e0 2024-09-21T02:07:07.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:07 smithi161 ceph-mon[34061]: pgmap v351: 129 pgs: 129 active+clean; 650 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:07.872+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fa0f006aff0 con 0x7fa0f01082e0 2024-09-21T02:07:08.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.181+0000 7fa0e67fc640 1 -- 172.21.15.16:0/2592317056 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7fa0e00ae0b0 con 0x7fa0f01082e0 2024-09-21T02:07:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 msgr2=0x7fa0dc0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 0x7fa0dc0b8af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec006010 tx=0x7fa0ec009680 comp rx=0 tx=0).stop 2024-09-21T02:07:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 msgr2=0x7fa0f019b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c8c0 tx=0x7fa0e000cd80 comp rx=0 tx=0).stop 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 shutdown_connections 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa0dc0b65e0 0x7fa0dc0b8af0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0f0108c30 0x7fa0f019bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f01082e0 0x7fa0f019b710 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 --2- 172.21.15.16:0/2592317056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01029b0 0x7fa0f019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 >> 172.21.15.16:0/2592317056 conn(0x7fa0f007a740 msgr2=0x7fa0f0106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 shutdown_connections 2024-09-21T02:07:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.184+0000 7fa0f732b640 1 -- 172.21.15.16:0/2592317056 wait complete. 2024-09-21T02:07:08.196 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-21T02:07:08.207 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T02:07:08.230 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T02:07:08.230 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:07:08.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-21T02:07:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 -- 172.21.15.16:0/3057791276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 msgr2=0x7f17ec1105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 --2- 172.21.15.16:0/3057791276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec1105b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009ca0 tx=0x7f17e402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 -- 172.21.15.16:0/3057791276 shutdown_connections 2024-09-21T02:07:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 --2- 172.21.15.16:0/3057791276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec1105b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 --2- 172.21.15.16:0/3057791276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec109430 0x7f17ec109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 --2- 172.21.15.16:0/3057791276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec103b00 0x7f17ec103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 -- 172.21.15.16:0/3057791276 >> 172.21.15.16:0/3057791276 conn(0x7f17ec0ff500 msgr2=0x7f17ec101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 -- 172.21.15.16:0/3057791276 shutdown_connections 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 -- 172.21.15.16:0/3057791276 wait complete. 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.346+0000 7f17f3038640 1 Processor -- start 2024-09-21T02:07:08.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 -- start start 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec103b00 0x7f17ec19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec109430 0x7f17ec19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec113940 con 0x7f17ec109d80 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1137c0 con 0x7f17ec109430 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f3038640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec113ac0 con 0x7f17ec103b00 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57612/0 (socket says 172.21.15.16:57612) 2024-09-21T02:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 -- 172.21.15.16:0/4237118262 learned_addr learned my addr 172.21.15.16:0/4237118262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17e3fff640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec109430 0x7f17ec19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f0dd4640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec103b00 0x7f17ec19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 -- 172.21.15.16:0/4237118262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec103b00 msgr2=0x7f17ec19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec103b00 0x7f17ec19a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 -- 172.21.15.16:0/4237118262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec109430 msgr2=0x7f17ec19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec109430 0x7f17ec19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 -- 172.21.15.16:0/4237118262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec1a0310 con 0x7f17ec109d80 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.347+0000 7f17f15d5640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f17e4031040 tx=0x7f17e4003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.348+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4032c90 con 0x7f17ec109d80 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.348+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f17e4032e30 con 0x7f17ec109d80 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.348+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec1a05a0 con 0x7f17ec109d80 2024-09-21T02:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.348+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4030af0 con 0x7f17ec109d80 2024-09-21T02:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.348+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1a6d90 con 0x7f17ec109d80 2024-09-21T02:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.349+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ec103f10 con 0x7f17ec109d80 2024-09-21T02:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.350+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f17e4030c90 con 0x7f17ec109d80 2024-09-21T02:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.351+0000 7f17e1ffb640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 0x7f17d40b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.351+0000 7f17f0dd4640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 0x7f17d40b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.351+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f17e4108a50 con 0x7f17ec109d80 2024-09-21T02:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.352+0000 7f17f0dd4640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 0x7f17d40b8af0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17e8005fd0 tx=0x7f17e8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.354+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f17e40cee50 con 0x7f17ec109d80 2024-09-21T02:07:08.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:08 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T02:07:08.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.564+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f17ec063800 con 0x7f17d40b65e0 2024-09-21T02:07:08.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:08 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T02:07:08.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:08 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd={"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} : dispatch 2024-09-21T02:07:08.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.729+0000 7f17e1ffb640 1 -- 172.21.15.16:0/4237118262 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f17ec063800 con 0x7f17d40b65e0 2024-09-21T02:07:08.730 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path / 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 msgr2=0x7f17d40b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 0x7f17d40b8af0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17e8005fd0 tx=0x7f17e8005850 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 msgr2=0x7f17ec19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f17e4031040 tx=0x7f17e4003f80 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 shutdown_connections 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f17d40b65e0 0x7f17d40b8af0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17ec109d80 0x7f17ec19fc10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17ec109430 0x7f17ec19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 --2- 172.21.15.16:0/4237118262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17ec103b00 0x7f17ec19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 >> 172.21.15.16:0/4237118262 conn(0x7f17ec0ff500 msgr2=0x7f17ec108a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 shutdown_connections 2024-09-21T02:07:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.732+0000 7f17f3038640 1 -- 172.21.15.16:0/4237118262 wait complete. 2024-09-21T02:07:08.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2235157740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 msgr2=0x7f7d74106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2235157740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d74106610 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009900 tx=0x7f7d6802a410 comp rx=0 tx=0).stop 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2235157740 shutdown_connections 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2235157740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 0x7f7d74110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2235157740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d74106610 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2235157740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 0x7f7d741024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2235157740 >> 172.21.15.16:0/2235157740 conn(0x7f7d74079ec0 msgr2=0x7f7d7407a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2235157740 shutdown_connections 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.859+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2235157740 wait complete. 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.860+0000 7f7d7a74a640 1 Processor -- start 2024-09-21T02:07:08.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.860+0000 7f7d7a74a640 1 -- start start 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 0x7f7d7419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 0x7f7d7419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7d741138d0 con 0x7f7d74106200 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d74113750 con 0x7f7d74106b50 2024-09-21T02:07:08.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7d74113a50 con 0x7f7d74102090 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57648/0 (socket says 172.21.15.16:57648) 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 -- 172.21.15.16:0/2099731706 learned_addr learned my addr 172.21.15.16:0/2099731706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d78ce7640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 0x7f7d7419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d73fff640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 0x7f7d7419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 -- 172.21.15.16:0/2099731706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 msgr2=0x7f7d7419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 0x7f7d7419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 -- 172.21.15.16:0/2099731706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 msgr2=0x7f7d7419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 0x7f7d7419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 -- 172.21.15.16:0/2099731706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d741a02a0 con 0x7f7d74106200 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d737fe640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7d68005940 tx=0x7f7d680040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6803f030 con 0x7f7d74106200 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d68003a80 con 0x7f7d74106200 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.861+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d741a0530 con 0x7f7d74106200 2024-09-21T02:07:08.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.862+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68035070 con 0x7f7d74106200 2024-09-21T02:07:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.862+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d741a6d20 con 0x7f7d74106200 2024-09-21T02:07:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.863+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7406b600 con 0x7f7d74106200 2024-09-21T02:07:08.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.864+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7f7d68042030 con 0x7f7d74106200 2024-09-21T02:07:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.865+0000 7f7d717fa640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 0x7f7d5c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.865+0000 7f7d73fff640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 0x7f7d5c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.865+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f7d68108180 con 0x7f7d74106200 2024-09-21T02:07:08.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.865+0000 7f7d73fff640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 0x7f7d5c0b8a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7d64005fd0 tx=0x7f7d64005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:08.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:08.868+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7d680ce520 con 0x7f7d74106200 2024-09-21T02:07:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.080+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f7d740638a0 con 0x7f7d5c0b6560 2024-09-21T02:07:09.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.143+0000 7f7d717fa640 1 -- 172.21.15.16:0/2099731706 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f7d740638a0 con 0x7f7d5c0b6560 2024-09-21T02:07:09.144 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path / 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 msgr2=0x7f7d5c0b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 0x7f7d5c0b8a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7d64005fd0 tx=0x7f7d64005850 comp rx=0 tx=0).stop 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 msgr2=0x7f7d7419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7d68005940 tx=0x7f7d680040e0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 shutdown_connections 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7d5c0b6560 0x7f7d5c0b8a70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d74106b50 0x7f7d7419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d74106200 0x7f7d7419a910 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 --2- 172.21.15.16:0/2099731706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d74102090 0x7f7d7419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 >> 172.21.15.16:0/2099731706 conn(0x7f7d74079ec0 msgr2=0x7f7d741095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 shutdown_connections 2024-09-21T02:07:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.145+0000 7f7d7a74a640 1 -- 172.21.15.16:0/2099731706 wait complete. 2024-09-21T02:07:09.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T02:07:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 -- 172.21.15.16:0/508436528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 msgr2=0x7ff29c10f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/508436528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c10f480 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff294009ca0 tx=0x7ff29402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 -- 172.21.15.16:0/508436528 shutdown_connections 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/508436528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c10f480 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/508436528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c108300 0x7ff29c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/508436528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c1029e0 0x7ff29c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 -- 172.21.15.16:0/508436528 >> 172.21.15.16:0/508436528 conn(0x7ff29c0784e0 msgr2=0x7ff29c0788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 -- 172.21.15.16:0/508436528 shutdown_connections 2024-09-21T02:07:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.271+0000 7ff2a3ede640 1 -- 172.21.15.16:0/508436528 wait complete. 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.272+0000 7ff2a3ede640 1 Processor -- start 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.272+0000 7ff2a3ede640 1 -- start start 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.272+0000 7ff2a3ede640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.272+0000 7ff2a3ede640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c108300 0x7ff29c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.272+0000 7ff2a3ede640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a3ede640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112810 con 0x7ff29c108c50 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a3ede640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112690 con 0x7ff29c108300 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a3ede640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112990 con 0x7ff29c1029e0 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a1c7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a1c7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:57450/0 (socket says 172.21.15.16:57450) 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a1c7a640 1 -- 172.21.15.16:0/2241872845 learned_addr learned my addr 172.21.15.16:0/2241872845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a1479640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c108300 0x7ff29c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 -- 172.21.15.16:0/2241872845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 msgr2=0x7ff29c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 -- 172.21.15.16:0/2241872845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c108300 msgr2=0x7ff29c19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c108300 0x7ff29c19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 -- 172.21.15.16:0/2241872845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c1a0210 con 0x7ff29c108c50 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a1c7a640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff2a247b640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c19fb10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff294031040 tx=0x7ff294003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.273+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294032c90 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.274+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff294032e30 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.274+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c1a04a0 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.274+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294030b20 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.274+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c1a6c90 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.274+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29c103c40 con 0x7ff29c108c50 2024-09-21T02:07:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.276+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7ff294030cc0 con 0x7ff29c108c50 2024-09-21T02:07:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.277+0000 7ff292ffd640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 0x7ff2880b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.277+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff294108a20 con 0x7ff29c108c50 2024-09-21T02:07:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.277+0000 7ff2a1c7a640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 0x7ff2880b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.277+0000 7ff2a1c7a640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 0x7ff2880b8a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff298011550 tx=0x7ff298005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:09.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.280+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff2940cee20 con 0x7ff29c108c50 2024-09-21T02:07:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:09 smithi016 ceph-mon[31245]: pgmap v352: 129 pgs: 129 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:09 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T02:07:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:09 smithi016 ceph-mon[31245]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:07:09.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:09 smithi016 ceph-mon[31245]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.490+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ff29c06b9c0 con 0x7ff2880b6560 2024-09-21T02:07:09.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.497+0000 7ff292ffd640 1 -- 172.21.15.16:0/2241872845 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7ff29c06b9c0 con 0x7ff2880b6560 2024-09-21T02:07:09.498 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T02:07:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 msgr2=0x7ff2880b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 0x7ff2880b8a70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff298011550 tx=0x7ff298005f90 comp rx=0 tx=0).stop 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 msgr2=0x7ff29c19fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c19fb10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff294031040 tx=0x7ff294003f80 comp rx=0 tx=0).stop 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 shutdown_connections 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2880b6560 0x7ff2880b8a70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c108c50 0x7ff29c19fb10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c108300 0x7ff29c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 --2- 172.21.15.16:0/2241872845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff29c1029e0 0x7ff29c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 >> 172.21.15.16:0/2241872845 conn(0x7ff29c0784e0 msgr2=0x7ff29c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 shutdown_connections 2024-09-21T02:07:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.499+0000 7ff2a3ede640 1 -- 172.21.15.16:0/2241872845 wait complete. 2024-09-21T02:07:09.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T02:07:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:09 smithi123 ceph-mon[33453]: pgmap v352: 129 pgs: 129 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:09 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T02:07:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:09 smithi123 ceph-mon[33453]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:07:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:09 smithi123 ceph-mon[33453]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 -- 172.21.15.16:0/521373110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c106b50 msgr2=0x7ff22c1105c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 --2- 172.21.15.16:0/521373110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c106b50 0x7ff22c1105c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff224009ca0 tx=0x7ff22402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 -- 172.21.15.16:0/521373110 shutdown_connections 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 --2- 172.21.15.16:0/521373110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c106b50 0x7ff22c1105c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 --2- 172.21.15.16:0/521373110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106200 0x7ff22c106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 --2- 172.21.15.16:0/521373110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c102090 0x7ff22c1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 -- 172.21.15.16:0/521373110 >> 172.21.15.16:0/521373110 conn(0x7ff22c079ec0 msgr2=0x7ff22c07a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 -- 172.21.15.16:0/521373110 shutdown_connections 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.625+0000 7ff232ddf640 1 -- 172.21.15.16:0/521373110 wait complete. 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 Processor -- start 2024-09-21T02:07:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 -- start start 2024-09-21T02:07:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c106200 0x7ff22c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106b50 0x7ff22c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c113920 con 0x7ff22c102090 2024-09-21T02:07:09.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1137a0 con 0x7ff22c106200 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.626+0000 7ff232ddf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c113aa0 con 0x7ff22c106b50 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57674/0 (socket says 172.21.15.16:57674) 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 -- 172.21.15.16:0/2024019050 learned_addr learned my addr 172.21.15.16:0/2024019050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff23137c640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106b50 0x7ff22c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff22bfff640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c106200 0x7ff22c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 -- 172.21.15.16:0/2024019050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106b50 msgr2=0x7ff22c19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106b50 0x7ff22c19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 -- 172.21.15.16:0/2024019050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c106200 msgr2=0x7ff22c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c106200 0x7ff22c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 -- 172.21.15.16:0/2024019050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1a02d0 con 0x7ff22c102090 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff230b7b640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff22000d950 tx=0x7ff22000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220017020 con 0x7ff22c102090 2024-09-21T02:07:09.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff220009c30 con 0x7ff22c102090 2024-09-21T02:07:09.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22000f620 con 0x7ff22c102090 2024-09-21T02:07:09.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.627+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1a05c0 con 0x7ff22c102090 2024-09-21T02:07:09.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.628+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c1a6d50 con 0x7ff22c102090 2024-09-21T02:07:09.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.628+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22c1024a0 con 0x7ff22c102090 2024-09-21T02:07:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.631+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) ==== 154529+0+0 (secure 0 0 0) 0x7ff22001a020 con 0x7ff22c102090 2024-09-21T02:07:09.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.632+0000 7ff229ffb640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 0x7ff21c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.632+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff2200e7410 con 0x7ff22c102090 2024-09-21T02:07:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.632+0000 7ff22bfff640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 0x7ff21c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:09.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.632+0000 7ff22bfff640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 0x7ff21c0b8af0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff22c10a190 tx=0x7ff218005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:09.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.635+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff220021070 con 0x7ff22c102090 2024-09-21T02:07:09.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:09 smithi161 ceph-mon[34061]: pgmap v352: 129 pgs: 129 active+clean; 651 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-21T02:07:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:09 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2592317056' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-21T02:07:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:09 smithi161 ceph-mon[34061]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-21T02:07:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:09 smithi161 ceph-mon[34061]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.846+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff22c063800 con 0x7ff21c0b65e0 2024-09-21T02:07:09.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.853+0000 7ff229ffb640 1 -- 172.21.15.16:0/2024019050 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff22c063800 con 0x7ff21c0b65e0 2024-09-21T02:07:09.854 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-09-21T02:07:09.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 msgr2=0x7ff21c0b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 0x7ff21c0b8af0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff22c10a190 tx=0x7ff218005f90 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 msgr2=0x7ff22c19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff22000d950 tx=0x7ff22000de10 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 shutdown_connections 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff21c0b65e0 0x7ff21c0b8af0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff22c106b50 0x7ff22c19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff22c106200 0x7ff22c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 --2- 172.21.15.16:0/2024019050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff22c102090 0x7ff22c19a3a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.855+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 >> 172.21.15.16:0/2024019050 conn(0x7ff22c079ec0 msgr2=0x7ff22c1084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.856+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 shutdown_connections 2024-09-21T02:07:09.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:09.856+0000 7ff232ddf640 1 -- 172.21.15.16:0/2024019050 wait complete. 2024-09-21T02:07:09.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-21T02:07:09.895 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-21T02:07:09.911 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-21T02:07:09.976 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T02:07:09.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:07:10.243 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:10.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:10 smithi016 ceph-mon[31245]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.491 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:10 smithi016 ceph-mon[31245]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:10 smithi123 ceph-mon[33453]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:10 smithi123 ceph-mon[33453]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:10 smithi161 ceph-mon[34061]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:10 smithi161 ceph-mon[34061]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/1113095067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 msgr2=0x7fcbac112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/1113095067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac112620 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009ca0 tx=0x7fcba402a950 comp rx=0 tx=0).stop 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/1113095067 shutdown_connections 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/1113095067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac112620 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/1113095067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102fa0 0x7fcbac10fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.717+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/1113095067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102650 0x7fcbac102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.718+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/1113095067 >> 172.21.15.16:0/1113095067 conn(0x7fcbac079760 msgr2=0x7fcbac079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.718+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/1113095067 shutdown_connections 2024-09-21T02:07:10.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.718+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/1113095067 wait complete. 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.718+0000 7fcbb3aa3640 1 Processor -- start 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.718+0000 7fcbb3aa3640 1 -- start start 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102650 0x7fcbac19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102fa0 0x7fcbac19cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac115b50 con 0x7fcbac110200 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1159d0 con 0x7fcbac102fa0 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb3aa3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac115cd0 con 0x7fcbac102650 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57694/0 (socket says 172.21.15.16:57694) 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 -- 172.21.15.16:0/955629885 learned_addr learned my addr 172.21.15.16:0/955629885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:10.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb103e640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102fa0 0x7fcbac19cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb183f640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102650 0x7fcbac19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 -- 172.21.15.16:0/955629885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102650 msgr2=0x7fcbac19c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102650 0x7fcbac19c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 -- 172.21.15.16:0/955629885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102fa0 msgr2=0x7fcbac19cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102fa0 0x7fcbac19cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 -- 172.21.15.16:0/955629885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac1a24f0 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.719+0000 7fcbb2040640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcba402ae70 tx=0x7fcba4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.720+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4034040 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.720+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcba4030850 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.720+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4004600 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.720+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbac1a2780 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.720+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbac1a8f40 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.721+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbac06c1f0 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.723+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fcba4037020 con 0x7fcbac110200 2024-09-21T02:07:10.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.723+0000 7fcba2ffd640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 0x7fcb980b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:10.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.723+0000 7fcbb183f640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 0x7fcb980b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:10.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.723+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fcba4108890 con 0x7fcbac110200 2024-09-21T02:07:10.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.724+0000 7fcbb183f640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 0x7fcb980b8b80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009b60 tx=0x7fcba8005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:10.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.727+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcba4083710 con 0x7fcbac110200 2024-09-21T02:07:10.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.914+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbac1093a0 con 0x7fcb980b6670 2024-09-21T02:07:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.920+0000 7fcba2ffd640 1 -- 172.21.15.16:0/955629885 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340819 (secure 0 0 0) 0x7fcbac1093a0 con 0x7fcb980b6670 2024-09-21T02:07:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 msgr2=0x7fcb980b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 0x7fcb980b8b80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009b60 tx=0x7fcba8005f90 comp rx=0 tx=0).stop 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 msgr2=0x7fcbac1a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcba402ae70 tx=0x7fcba4003ce0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 shutdown_connections 2024-09-21T02:07:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fcb980b6670 0x7fcb980b8b80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbac110200 0x7fcbac1a1dc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac102fa0 0x7fcbac19cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 --2- 172.21.15.16:0/955629885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbac102650 0x7fcbac19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 >> 172.21.15.16:0/955629885 conn(0x7fcbac079760 msgr2=0x7fcbac108c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 shutdown_connections 2024-09-21T02:07:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:10.924+0000 7fcbb3aa3640 1 -- 172.21.15.16:0/955629885 wait complete. 2024-09-21T02:07:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:11 smithi123 ceph-mon[33453]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:11 smithi123 ceph-mon[33453]: pgmap v353: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-21T02:07:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:11 smithi123 ceph-mon[33453]: mgrmap e23: x(active, since 9m), standbys: z, y 2024-09-21T02:07:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:11 smithi016 ceph-mon[31245]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:11 smithi016 ceph-mon[31245]: pgmap v353: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-21T02:07:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:11 smithi016 ceph-mon[31245]: mgrmap e23: x(active, since 9m), standbys: z, y 2024-09-21T02:07:11.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:11 smithi161 ceph-mon[34061]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:11.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:11 smithi161 ceph-mon[34061]: pgmap v353: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 62 KiB/s wr, 18 op/s 2024-09-21T02:07:11.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:11 smithi161 ceph-mon[34061]: mgrmap e23: x(active, since 9m), standbys: z, y 2024-09-21T02:07:11.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-21T02:07:10.059412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1061854,"num_objects":103,"num_object_clones":0,"num_object_copies":311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":1224,"num_read_kb":735,"num_write":458,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394224,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547600,"statfs":{"total":1151940427776,"available":1151536742400,"internally_reserved":0,"allocated":11976704,"data_stored":7973892,"data_compressed":465210,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":396241,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":153,"num_read_kb":87,"num_write":69,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003690"},"pg_stats":[{"pgid":"2.1f","version":"86'14","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:05.898888+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:07:05.898888+0000","last_peered":"2024-09-21T02:07:05.898888+0000","last_clean":"2024-09-21T02:07:05.898888+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:07:05.898888+0000","last_undegraded":"2024-09-21T02:07:05.898888+0000","last_fullsized":"2024-09-21T02:07:05.898888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435128+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:06:52.435128+0000","last_peered":"2024-09-21T02:06:52.435128+0000","last_clean":"2024-09-21T02:06:52.435128+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:06:52.435128+0000","last_undegraded":"2024-09-21T02:06:52.435128+0000","last_fullsized":"2024-09-21T02:06:52.435128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309551+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:05:54.309551+0000","last_peered":"2024-09-21T02:05:54.309551+0000","last_clean":"2024-09-21T02:05:54.309551+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:05:54.309551+0000","last_undegraded":"2024-09-21T02:05:54.309551+0000","last_fullsized":"2024-09-21T02:05:54.309551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694353+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:05:54.694353+0000","last_peered":"2024-09-21T02:05:54.694353+0000","last_clean":"2024-09-21T02:05:54.694353+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:05:54.694353+0000","last_undegraded":"2024-09-21T02:05:54.694353+0000","last_fullsized":"2024-09-21T02:05:54.694353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434557+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:06:52.434557+0000","last_peered":"2024-09-21T02:06:52.434557+0000","last_clean":"2024-09-21T02:06:52.434557+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:06:52.434557+0000","last_undegraded":"2024-09-21T02:06:52.434557+0000","last_fullsized":"2024-09-21T02:06:52.434557+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313872+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:05:54.313872+0000","last_peered":"2024-09-21T02:05:54.313872+0000","last_clean":"2024-09-21T02:05:54.313872+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:05:54.313872+0000","last_undegraded":"2024-09-21T02:05:54.313872+0000","last_fullsized":"2024-09-21T02:05:54.313872+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314180+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:54.314180+0000","last_peered":"2024-09-21T02:05:54.314180+0000","last_clean":"2024-09-21T02:05:54.314180+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:54.314180+0000","last_undegraded":"2024-09-21T02:05:54.314180+0000","last_fullsized":"2024-09-21T02:05:54.314180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447667+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:06:52.447667+0000","last_peered":"2024-09-21T02:06:52.447667+0000","last_clean":"2024-09-21T02:06:52.447667+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:06:52.447667+0000","last_undegraded":"2024-09-21T02:06:52.447667+0000","last_fullsized":"2024-09-21T02:06:52.447667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841186+0000","last_change":"2024-09-21T02:05:54.841083+0000","last_active":"2024-09-21T02:05:54.841186+0000","last_peered":"2024-09-21T02:05:54.841186+0000","last_clean":"2024-09-21T02:05:54.841186+0000","last_became_active":"2024-09-21T02:05:54.840645+0000","last_became_peered":"2024-09-21T02:05:54.840645+0000","last_unstale":"2024-09-21T02:05:54.841186+0000","last_undegraded":"2024-09-21T02:05:54.841186+0000","last_fullsized":"2024-09-21T02:05:54.841186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:33:35.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840304+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:54.840304+0000","last_peered":"2024-09-21T02:05:54.840304+0000","last_clean":"2024-09-21T02:05:54.840304+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:54.840304+0000","last_undegraded":"2024-09-21T02:05:54.840304+0000","last_fullsized":"2024-09-21T02:05:54.840304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445936+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:06:52.445936+0000","last_peered":"2024-09-21T02:06:52.445936+0000","last_clean":"2024-09-21T02:06:52.445936+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:06:52.445936+0000","last_undegraded":"2024-09-21T02:06:52.445936+0000","last_fullsized":"2024-09-21T02:06:52.445936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313927+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:05:54.313927+0000","last_peered":"2024-09-21T02:05:54.313927+0000","last_clean":"2024-09-21T02:05:54.313927+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:05:54.313927+0000","last_undegraded":"2024-09-21T02:05:54.313927+0000","last_fullsized":"2024-09-21T02:05:54.313927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445821+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:06:52.445821+0000","last_peered":"2024-09-21T02:06:52.445821+0000","last_clean":"2024-09-21T02:06:52.445821+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:06:52.445821+0000","last_undegraded":"2024-09-21T02:06:52.445821+0000","last_fullsized":"2024-09-21T02:06:52.445821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":56,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839948+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:54.839948+0000","last_peered":"2024-09-21T02:05:54.839948+0000","last_clean":"2024-09-21T02:05:54.839948+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:54.839948+0000","last_undegraded":"2024-09-21T02:05:54.839948+0000","last_fullsized":"2024-09-21T02:05:54.839948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314223+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:05:54.314223+0000","last_peered":"2024-09-21T02:05:54.314223+0000","last_clean":"2024-09-21T02:05:54.314223+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:05:54.314223+0000","last_undegraded":"2024-09-21T02:05:54.314223+0000","last_fullsized":"2024-09-21T02:05:54.314223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432917+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:06:52.432917+0000","last_peered":"2024-09-21T02:06:52.432917+0000","last_clean":"2024-09-21T02:06:52.432917+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:06:52.432917+0000","last_undegraded":"2024-09-21T02:06:52.432917+0000","last_fullsized":"2024-09-21T02:06:52.432917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309914+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:05:54.309914+0000","last_peered":"2024-09-21T02:05:54.309914+0000","last_clean":"2024-09-21T02:05:54.309914+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:05:54.309914+0000","last_undegraded":"2024-09-21T02:05:54.309914+0000","last_fullsized":"2024-09-21T02:05:54.309914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314252+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:05:54.314252+0000","last_peered":"2024-09-21T02:05:54.314252+0000","last_clean":"2024-09-21T02:05:54.314252+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:05:54.314252+0000","last_undegraded":"2024-09-21T02:05:54.314252+0000","last_fullsized":"2024-09-21T02:05:54.314252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428361+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.428361+0000","last_peered":"2024-09-21T02:06:52.428361+0000","last_clean":"2024-09-21T02:06:52.428361+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:06:52.428361+0000","last_undegraded":"2024-09-21T02:06:52.428361+0000","last_fullsized":"2024-09-21T02:06:52.428361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693914+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:54.693914+0000","last_peered":"2024-09-21T02:05:54.693914+0000","last_clean":"2024-09-21T02:05:54.693914+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:54.693914+0000","last_undegraded":"2024-09-21T02:05:54.693914+0000","last_fullsized":"2024-09-21T02:05:54.693914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.754062+0000","last_change":"2024-09-21T02:05:54.753929+0000","last_active":"2024-09-21T02:05:54.754062+0000","last_peered":"2024-09-21T02:05:54.754062+0000","last_clean":"2024-09-21T02:05:54.754062+0000","last_became_active":"2024-09-21T02:05:54.753431+0000","last_became_peered":"2024-09-21T02:05:54.753431+0000","last_unstale":"2024-09-21T02:05:54.754062+0000","last_undegraded":"2024-09-21T02:05:54.754062+0000","last_fullsized":"2024-09-21T02:05:54.754062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:20:41.311972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.391580+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:06:52.391580+0000","last_peered":"2024-09-21T02:06:52.391580+0000","last_clean":"2024-09-21T02:06:52.391580+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:06:52.391580+0000","last_undegraded":"2024-09-21T02:06:52.391580+0000","last_fullsized":"2024-09-21T02:06:52.391580+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839098+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:54.839098+0000","last_peered":"2024-09-21T02:05:54.839098+0000","last_clean":"2024-09-21T02:05:54.839098+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:54.839098+0000","last_undegraded":"2024-09-21T02:05:54.839098+0000","last_fullsized":"2024-09-21T02:05:54.839098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314296+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:05:54.314296+0000","last_peered":"2024-09-21T02:05:54.314296+0000","last_clean":"2024-09-21T02:05:54.314296+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:05:54.314296+0000","last_undegraded":"2024-09-21T02:05:54.314296+0000","last_fullsized":"2024-09-21T02:05:54.314296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432105+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:06:52.432105+0000","last_peered":"2024-09-21T02:06:52.432105+0000","last_clean":"2024-09-21T02:06:52.432105+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:06:52.432105+0000","last_undegraded":"2024-09-21T02:06:52.432105+0000","last_fullsized":"2024-09-21T02:06:52.432105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314615+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:54.314615+0000","last_peered":"2024-09-21T02:05:54.314615+0000","last_clean":"2024-09-21T02:05:54.314615+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:54.314615+0000","last_undegraded":"2024-09-21T02:05:54.314615+0000","last_fullsized":"2024-09-21T02:05:54.314615+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309689+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:05:54.309689+0000","last_peered":"2024-09-21T02:05:54.309689+0000","last_clean":"2024-09-21T02:05:54.309689+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:05:54.309689+0000","last_undegraded":"2024-09-21T02:05:54.309689+0000","last_fullsized":"2024-09-21T02:05:54.309689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437884+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:06:52.437884+0000","last_peered":"2024-09-21T02:06:52.437884+0000","last_clean":"2024-09-21T02:06:52.437884+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:06:52.437884+0000","last_undegraded":"2024-09-21T02:06:52.437884+0000","last_fullsized":"2024-09-21T02:06:52.437884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'18","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.675186+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:06:52.675186+0000","last_peered":"2024-09-21T02:06:52.675186+0000","last_clean":"2024-09-21T02:06:52.675186+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:06:52.675186+0000","last_undegraded":"2024-09-21T02:06:52.675186+0000","last_fullsized":"2024-09-21T02:06:52.675186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693575+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:05:54.693575+0000","last_peered":"2024-09-21T02:05:54.693575+0000","last_clean":"2024-09-21T02:05:54.693575+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:05:54.693575+0000","last_undegraded":"2024-09-21T02:05:54.693575+0000","last_fullsized":"2024-09-21T02:05:54.693575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430290+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:06:52.430290+0000","last_peered":"2024-09-21T02:06:52.430290+0000","last_clean":"2024-09-21T02:06:52.430290+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:06:52.430290+0000","last_undegraded":"2024-09-21T02:06:52.430290+0000","last_fullsized":"2024-09-21T02:06:52.430290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314156+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:54.314156+0000","last_peered":"2024-09-21T02:05:54.314156+0000","last_clean":"2024-09-21T02:05:54.314156+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:54.314156+0000","last_undegraded":"2024-09-21T02:05:54.314156+0000","last_fullsized":"2024-09-21T02:05:54.314156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314000+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:05:54.314000+0000","last_peered":"2024-09-21T02:05:54.314000+0000","last_clean":"2024-09-21T02:05:54.314000+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:05:54.314000+0000","last_undegraded":"2024-09-21T02:05:54.314000+0000","last_fullsized":"2024-09-21T02:05:54.314000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431581+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.431581+0000","last_peered":"2024-09-21T02:06:52.431581+0000","last_clean":"2024-09-21T02:06:52.431581+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:06:52.431581+0000","last_undegraded":"2024-09-21T02:06:52.431581+0000","last_fullsized":"2024-09-21T02:06:52.431581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796090+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:05:54.796090+0000","last_peered":"2024-09-21T02:05:54.796090+0000","last_clean":"2024-09-21T02:05:54.796090+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:05:54.796090+0000","last_undegraded":"2024-09-21T02:05:54.796090+0000","last_fullsized":"2024-09-21T02:05:54.796090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839552+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:05:54.839552+0000","last_peered":"2024-09-21T02:05:54.839552+0000","last_clean":"2024-09-21T02:05:54.839552+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:05:54.839552+0000","last_undegraded":"2024-09-21T02:05:54.839552+0000","last_fullsized":"2024-09-21T02:05:54.839552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427901+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:06:52.427901+0000","last_peered":"2024-09-21T02:06:52.427901+0000","last_clean":"2024-09-21T02:06:52.427901+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:06:52.427901+0000","last_undegraded":"2024-09-21T02:06:52.427901+0000","last_fullsized":"2024-09-21T02:06:52.427901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'8","reported_seq":156,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:04.898259+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:07:04.898259+0000","last_peered":"2024-09-21T02:07:04.898259+0000","last_clean":"2024-09-21T02:07:04.898259+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:07:04.898259+0000","last_undegraded":"2024-09-21T02:07:04.898259+0000","last_fullsized":"2024-09-21T02:07:04.898259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308834+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:05:54.308834+0000","last_peered":"2024-09-21T02:05:54.308834+0000","last_clean":"2024-09-21T02:05:54.308834+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:05:54.308834+0000","last_undegraded":"2024-09-21T02:05:54.308834+0000","last_fullsized":"2024-09-21T02:05:54.308834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431202+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:06:52.431202+0000","last_peered":"2024-09-21T02:06:52.431202+0000","last_clean":"2024-09-21T02:06:52.431202+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:06:52.431202+0000","last_undegraded":"2024-09-21T02:06:52.431202+0000","last_fullsized":"2024-09-21T02:06:52.431202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":152,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:03.896273+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:07:03.896273+0000","last_peered":"2024-09-21T02:07:03.896273+0000","last_clean":"2024-09-21T02:07:03.896273+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:07:03.896273+0000","last_undegraded":"2024-09-21T02:07:03.896273+0000","last_fullsized":"2024-09-21T02:07:03.896273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309231+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:05:54.309231+0000","last_peered":"2024-09-21T02:05:54.309231+0000","last_clean":"2024-09-21T02:05:54.309231+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:05:54.309231+0000","last_undegraded":"2024-09-21T02:05:54.309231+0000","last_fullsized":"2024-09-21T02:05:54.309231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431015+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:06:52.431015+0000","last_peered":"2024-09-21T02:06:52.431015+0000","last_clean":"2024-09-21T02:06:52.431015+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:06:52.431015+0000","last_undegraded":"2024-09-21T02:06:52.431015+0000","last_fullsized":"2024-09-21T02:06:52.431015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799870+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:54.799870+0000","last_peered":"2024-09-21T02:05:54.799870+0000","last_clean":"2024-09-21T02:05:54.799870+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:54.799870+0000","last_undegraded":"2024-09-21T02:05:54.799870+0000","last_fullsized":"2024-09-21T02:05:54.799870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752633+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:05:54.752633+0000","last_peered":"2024-09-21T02:05:54.752633+0000","last_clean":"2024-09-21T02:05:54.752633+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:05:54.752633+0000","last_undegraded":"2024-09-21T02:05:54.752633+0000","last_fullsized":"2024-09-21T02:05:54.752633+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436058+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:06:52.436058+0000","last_peered":"2024-09-21T02:06:52.436058+0000","last_clean":"2024-09-21T02:06:52.436058+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:06:52.436058+0000","last_undegraded":"2024-09-21T02:06:52.436058+0000","last_fullsized":"2024-09-21T02:06:52.436058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314486+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:54.314486+0000","last_peered":"2024-09-21T02:05:54.314486+0000","last_clean":"2024-09-21T02:05:54.314486+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:54.314486+0000","last_undegraded":"2024-09-21T02:05:54.314486+0000","last_fullsized":"2024-09-21T02:05:54.314486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799508+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:05:54.799508+0000","last_peered":"2024-09-21T02:05:54.799508+0000","last_clean":"2024-09-21T02:05:54.799508+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:05:54.799508+0000","last_undegraded":"2024-09-21T02:05:54.799508+0000","last_fullsized":"2024-09-21T02:05:54.799508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799546+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:05:54.799546+0000","last_peered":"2024-09-21T02:05:54.799546+0000","last_clean":"2024-09-21T02:05:54.799546+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:05:54.799546+0000","last_undegraded":"2024-09-21T02:05:54.799546+0000","last_fullsized":"2024-09-21T02:05:54.799546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444281+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:06:52.444281+0000","last_peered":"2024-09-21T02:06:52.444281+0000","last_clean":"2024-09-21T02:06:52.444281+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:06:52.444281+0000","last_undegraded":"2024-09-21T02:06:52.444281+0000","last_fullsized":"2024-09-21T02:06:52.444281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'8","reported_seq":159,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:05.913107+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:07:05.913107+0000","last_peered":"2024-09-21T02:07:05.913107+0000","last_clean":"2024-09-21T02:07:05.913107+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:07:05.913107+0000","last_undegraded":"2024-09-21T02:07:05.913107+0000","last_fullsized":"2024-09-21T02:07:05.913107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838953+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:05:54.838953+0000","last_peered":"2024-09-21T02:05:54.838953+0000","last_clean":"2024-09-21T02:05:54.838953+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:05:54.838953+0000","last_undegraded":"2024-09-21T02:05:54.838953+0000","last_fullsized":"2024-09-21T02:05:54.838953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435516+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:06:52.435516+0000","last_peered":"2024-09-21T02:06:52.435516+0000","last_clean":"2024-09-21T02:06:52.435516+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:06:52.435516+0000","last_undegraded":"2024-09-21T02:06:52.435516+0000","last_fullsized":"2024-09-21T02:06:52.435516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'54","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.723869+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:07:08.723869+0000","last_peered":"2024-09-21T02:07:08.723869+0000","last_clean":"2024-09-21T02:07:08.723869+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:07:08.723869+0000","last_undegraded":"2024-09-21T02:07:08.723869+0000","last_fullsized":"2024-09-21T02:07:08.723869+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":63,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838590+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:05:54.838590+0000","last_peered":"2024-09-21T02:05:54.838590+0000","last_clean":"2024-09-21T02:05:54.838590+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:05:54.838590+0000","last_undegraded":"2024-09-21T02:05:54.838590+0000","last_fullsized":"2024-09-21T02:05:54.838590+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.443855+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:06:52.443855+0000","last_peered":"2024-09-21T02:06:52.443855+0000","last_clean":"2024-09-21T02:06:52.443855+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:06:52.443855+0000","last_undegraded":"2024-09-21T02:06:52.443855+0000","last_fullsized":"2024-09-21T02:06:52.443855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309244+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:54.309244+0000","last_peered":"2024-09-21T02:05:54.309244+0000","last_clean":"2024-09-21T02:05:54.309244+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:54.309244+0000","last_undegraded":"2024-09-21T02:05:54.309244+0000","last_fullsized":"2024-09-21T02:05:54.309244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751778+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:05:54.751778+0000","last_peered":"2024-09-21T02:05:54.751778+0000","last_clean":"2024-09-21T02:05:54.751778+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:05:54.751778+0000","last_undegraded":"2024-09-21T02:05:54.751778+0000","last_fullsized":"2024-09-21T02:05:54.751778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436869+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:06:52.436869+0000","last_peered":"2024-09-21T02:06:52.436869+0000","last_clean":"2024-09-21T02:06:52.436869+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:06:52.436869+0000","last_undegraded":"2024-09-21T02:06:52.436869+0000","last_fullsized":"2024-09-21T02:06:52.436869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840384+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:05:54.840384+0000","last_peered":"2024-09-21T02:05:54.840384+0000","last_clean":"2024-09-21T02:05:54.840384+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:05:54.840384+0000","last_undegraded":"2024-09-21T02:05:54.840384+0000","last_fullsized":"2024-09-21T02:05:54.840384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751987+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:05:54.751987+0000","last_peered":"2024-09-21T02:05:54.751987+0000","last_clean":"2024-09-21T02:05:54.751987+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:05:54.751987+0000","last_undegraded":"2024-09-21T02:05:54.751987+0000","last_fullsized":"2024-09-21T02:05:54.751987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437619+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:06:52.437619+0000","last_peered":"2024-09-21T02:06:52.437619+0000","last_clean":"2024-09-21T02:06:52.437619+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:06:52.437619+0000","last_undegraded":"2024-09-21T02:06:52.437619+0000","last_fullsized":"2024-09-21T02:06:52.437619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839608+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:54.839608+0000","last_peered":"2024-09-21T02:05:54.839608+0000","last_clean":"2024-09-21T02:05:54.839608+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:54.839608+0000","last_undegraded":"2024-09-21T02:05:54.839608+0000","last_fullsized":"2024-09-21T02:05:54.839608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309217+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:05:54.309217+0000","last_peered":"2024-09-21T02:05:54.309217+0000","last_clean":"2024-09-21T02:05:54.309217+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:05:54.309217+0000","last_undegraded":"2024-09-21T02:05:54.309217+0000","last_fullsized":"2024-09-21T02:05:54.309217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445290+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:06:52.445290+0000","last_peered":"2024-09-21T02:06:52.445290+0000","last_clean":"2024-09-21T02:06:52.445290+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:06:52.445290+0000","last_undegraded":"2024-09-21T02:06:52.445290+0000","last_fullsized":"2024-09-21T02:06:52.445290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'8","reported_seq":161,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:06.902308+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:07:06.902308+0000","last_peered":"2024-09-21T02:07:06.902308+0000","last_clean":"2024-09-21T02:07:06.902308+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:07:06.902308+0000","last_undegraded":"2024-09-21T02:07:06.902308+0000","last_fullsized":"2024-09-21T02:07:06.902308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796399+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:05:54.796399+0000","last_peered":"2024-09-21T02:05:54.796399+0000","last_clean":"2024-09-21T02:05:54.796399+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:05:54.796399+0000","last_undegraded":"2024-09-21T02:05:54.796399+0000","last_fullsized":"2024-09-21T02:05:54.796399+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446922+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:06:52.446922+0000","last_peered":"2024-09-21T02:06:52.446922+0000","last_clean":"2024-09-21T02:06:52.446922+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:06:52.446922+0000","last_undegraded":"2024-09-21T02:06:52.446922+0000","last_fullsized":"2024-09-21T02:06:52.446922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752428+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:54.752428+0000","last_peered":"2024-09-21T02:05:54.752428+0000","last_clean":"2024-09-21T02:05:54.752428+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:54.752428+0000","last_undegraded":"2024-09-21T02:05:54.752428+0000","last_fullsized":"2024-09-21T02:05:54.752428+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314337+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:05:54.314337+0000","last_peered":"2024-09-21T02:05:54.314337+0000","last_clean":"2024-09-21T02:05:54.314337+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:05:54.314337+0000","last_undegraded":"2024-09-21T02:05:54.314337+0000","last_fullsized":"2024-09-21T02:05:54.314337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446213+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:06:52.446213+0000","last_peered":"2024-09-21T02:06:52.446213+0000","last_clean":"2024-09-21T02:06:52.446213+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:06:52.446213+0000","last_undegraded":"2024-09-21T02:06:52.446213+0000","last_fullsized":"2024-09-21T02:06:52.446213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309963+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:54.309963+0000","last_peered":"2024-09-21T02:05:54.309963+0000","last_clean":"2024-09-21T02:05:54.309963+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:54.309963+0000","last_undegraded":"2024-09-21T02:05:54.309963+0000","last_fullsized":"2024-09-21T02:05:54.309963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752780+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:05:54.752780+0000","last_peered":"2024-09-21T02:05:54.752780+0000","last_clean":"2024-09-21T02:05:54.752780+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:05:54.752780+0000","last_undegraded":"2024-09-21T02:05:54.752780+0000","last_fullsized":"2024-09-21T02:05:54.752780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.424275+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:06:52.424275+0000","last_peered":"2024-09-21T02:06:52.424275+0000","last_clean":"2024-09-21T02:06:52.424275+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:06:52.424275+0000","last_undegraded":"2024-09-21T02:06:52.424275+0000","last_fullsized":"2024-09-21T02:06:52.424275+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'38","reported_seq":305,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.920653+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:07:07.920653+0000","last_peered":"2024-09-21T02:07:07.920653+0000","last_clean":"2024-09-21T02:07:07.920653+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:07:07.920653+0000","last_undegraded":"2024-09-21T02:07:07.920653+0000","last_fullsized":"2024-09-21T02:07:07.920653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66954,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":227,"num_read_kb":179,"num_write":40,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795725+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:05:54.795725+0000","last_peered":"2024-09-21T02:05:54.795725+0000","last_clean":"2024-09-21T02:05:54.795725+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:05:54.795725+0000","last_undegraded":"2024-09-21T02:05:54.795725+0000","last_fullsized":"2024-09-21T02:05:54.795725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444121+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:06:52.444121+0000","last_peered":"2024-09-21T02:06:52.444121+0000","last_clean":"2024-09-21T02:06:52.444121+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:06:52.444121+0000","last_undegraded":"2024-09-21T02:06:52.444121+0000","last_fullsized":"2024-09-21T02:06:52.444121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752935+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:54.752935+0000","last_peered":"2024-09-21T02:05:54.752935+0000","last_clean":"2024-09-21T02:05:54.752935+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:54.752935+0000","last_undegraded":"2024-09-21T02:05:54.752935+0000","last_fullsized":"2024-09-21T02:05:54.752935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314193+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:05:54.314193+0000","last_peered":"2024-09-21T02:05:54.314193+0000","last_clean":"2024-09-21T02:05:54.314193+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:05:54.314193+0000","last_undegraded":"2024-09-21T02:05:54.314193+0000","last_fullsized":"2024-09-21T02:05:54.314193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437003+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:06:52.437003+0000","last_peered":"2024-09-21T02:06:52.437003+0000","last_clean":"2024-09-21T02:06:52.437003+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:06:52.437003+0000","last_undegraded":"2024-09-21T02:06:52.437003+0000","last_fullsized":"2024-09-21T02:06:52.437003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799684+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:54.799684+0000","last_peered":"2024-09-21T02:05:54.799684+0000","last_clean":"2024-09-21T02:05:54.799684+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:54.799684+0000","last_undegraded":"2024-09-21T02:05:54.799684+0000","last_fullsized":"2024-09-21T02:05:54.799684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839390+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:05:54.839390+0000","last_peered":"2024-09-21T02:05:54.839390+0000","last_clean":"2024-09-21T02:05:54.839390+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:05:54.839390+0000","last_undegraded":"2024-09-21T02:05:54.839390+0000","last_fullsized":"2024-09-21T02:05:54.839390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447850+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:06:52.447850+0000","last_peered":"2024-09-21T02:06:52.447850+0000","last_clean":"2024-09-21T02:06:52.447850+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:06:52.447850+0000","last_undegraded":"2024-09-21T02:06:52.447850+0000","last_fullsized":"2024-09-21T02:06:52.447850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'6","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.903683+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:07:07.903683+0000","last_peered":"2024-09-21T02:07:07.903683+0000","last_clean":"2024-09-21T02:07:07.903683+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:07:07.903683+0000","last_undegraded":"2024-09-21T02:07:07.903683+0000","last_fullsized":"2024-09-21T02:07:07.903683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693641+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:05:54.693641+0000","last_peered":"2024-09-21T02:05:54.693641+0000","last_clean":"2024-09-21T02:05:54.693641+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:05:54.693641+0000","last_undegraded":"2024-09-21T02:05:54.693641+0000","last_fullsized":"2024-09-21T02:05:54.693641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838683+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:05:54.838683+0000","last_peered":"2024-09-21T02:05:54.838683+0000","last_clean":"2024-09-21T02:05:54.838683+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:05:54.838683+0000","last_undegraded":"2024-09-21T02:05:54.838683+0000","last_fullsized":"2024-09-21T02:05:54.838683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429092+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:06:52.429092+0000","last_peered":"2024-09-21T02:06:52.429092+0000","last_clean":"2024-09-21T02:06:52.429092+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:06:52.429092+0000","last_undegraded":"2024-09-21T02:06:52.429092+0000","last_fullsized":"2024-09-21T02:06:52.429092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752288+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:54.752288+0000","last_peered":"2024-09-21T02:05:54.752288+0000","last_clean":"2024-09-21T02:05:54.752288+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:54.752288+0000","last_undegraded":"2024-09-21T02:05:54.752288+0000","last_fullsized":"2024-09-21T02:05:54.752288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.842081+0000","last_change":"2024-09-21T02:05:54.841426+0000","last_active":"2024-09-21T02:05:54.842081+0000","last_peered":"2024-09-21T02:05:54.842081+0000","last_clean":"2024-09-21T02:05:54.842081+0000","last_became_active":"2024-09-21T02:05:54.840855+0000","last_became_peered":"2024-09-21T02:05:54.840855+0000","last_unstale":"2024-09-21T02:05:54.842081+0000","last_undegraded":"2024-09-21T02:05:54.842081+0000","last_fullsized":"2024-09-21T02:05:54.842081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:40.637427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438022+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:06:52.438022+0000","last_peered":"2024-09-21T02:06:52.438022+0000","last_clean":"2024-09-21T02:06:52.438022+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:06:52.438022+0000","last_undegraded":"2024-09-21T02:06:52.438022+0000","last_fullsized":"2024-09-21T02:06:52.438022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308608+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:54.308608+0000","last_peered":"2024-09-21T02:05:54.308608+0000","last_clean":"2024-09-21T02:05:54.308608+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:54.308608+0000","last_undegraded":"2024-09-21T02:05:54.308608+0000","last_fullsized":"2024-09-21T02:05:54.308608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309526+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:54.309526+0000","last_peered":"2024-09-21T02:05:54.309526+0000","last_clean":"2024-09-21T02:05:54.309526+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:54.309526+0000","last_undegraded":"2024-09-21T02:05:54.309526+0000","last_fullsized":"2024-09-21T02:05:54.309526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437429+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:06:52.437429+0000","last_peered":"2024-09-21T02:06:52.437429+0000","last_clean":"2024-09-21T02:06:52.437429+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:06:52.437429+0000","last_undegraded":"2024-09-21T02:06:52.437429+0000","last_fullsized":"2024-09-21T02:06:52.437429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309752+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:05:54.309752+0000","last_peered":"2024-09-21T02:05:54.309752+0000","last_clean":"2024-09-21T02:05:54.309752+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:05:54.309752+0000","last_undegraded":"2024-09-21T02:05:54.309752+0000","last_fullsized":"2024-09-21T02:05:54.309752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'8","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.904253+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:07:07.904253+0000","last_peered":"2024-09-21T02:07:07.904253+0000","last_clean":"2024-09-21T02:07:07.904253+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:07:07.904253+0000","last_undegraded":"2024-09-21T02:07:07.904253+0000","last_fullsized":"2024-09-21T02:07:07.904253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.420899+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:06:52.420899+0000","last_peered":"2024-09-21T02:06:52.420899+0000","last_clean":"2024-09-21T02:06:52.420899+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:06:52.420899+0000","last_undegraded":"2024-09-21T02:06:52.420899+0000","last_fullsized":"2024-09-21T02:06:52.420899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309832+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:05:54.309832+0000","last_peered":"2024-09-21T02:05:54.309832+0000","last_clean":"2024-09-21T02:05:54.309832+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:05:54.309832+0000","last_undegraded":"2024-09-21T02:05:54.309832+0000","last_fullsized":"2024-09-21T02:05:54.309832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437860+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:06:52.437860+0000","last_peered":"2024-09-21T02:06:52.437860+0000","last_clean":"2024-09-21T02:06:52.437860+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:06:52.437860+0000","last_undegraded":"2024-09-21T02:06:52.437860+0000","last_fullsized":"2024-09-21T02:06:52.437860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435988+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:06:52.435988+0000","last_peered":"2024-09-21T02:06:52.435988+0000","last_clean":"2024-09-21T02:06:52.435988+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:06:52.435988+0000","last_undegraded":"2024-09-21T02:06:52.435988+0000","last_fullsized":"2024-09-21T02:06:52.435988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427485+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:06:52.427485+0000","last_peered":"2024-09-21T02:06:52.427485+0000","last_clean":"2024-09-21T02:06:52.427485+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:06:52.427485+0000","last_undegraded":"2024-09-21T02:06:52.427485+0000","last_fullsized":"2024-09-21T02:06:52.427485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435191+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:06:52.435191+0000","last_peered":"2024-09-21T02:06:52.435191+0000","last_clean":"2024-09-21T02:06:52.435191+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:06:52.435191+0000","last_undegraded":"2024-09-21T02:06:52.435191+0000","last_fullsized":"2024-09-21T02:06:52.435191+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431472+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:06:52.431472+0000","last_peered":"2024-09-21T02:06:52.431472+0000","last_clean":"2024-09-21T02:06:52.431472+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:06:52.431472+0000","last_undegraded":"2024-09-21T02:06:52.431472+0000","last_fullsized":"2024-09-21T02:06:52.431472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436873+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:06:52.436873+0000","last_peered":"2024-09-21T02:06:52.436873+0000","last_clean":"2024-09-21T02:06:52.436873+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:06:52.436873+0000","last_undegraded":"2024-09-21T02:06:52.436873+0000","last_fullsized":"2024-09-21T02:06:52.436873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.421562+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:06:52.421562+0000","last_peered":"2024-09-21T02:06:52.421562+0000","last_clean":"2024-09-21T02:06:52.421562+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:06:52.421562+0000","last_undegraded":"2024-09-21T02:06:52.421562+0000","last_fullsized":"2024-09-21T02:06:52.421562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429068+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:06:52.429068+0000","last_peered":"2024-09-21T02:06:52.429068+0000","last_clean":"2024-09-21T02:06:52.429068+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:06:52.429068+0000","last_undegraded":"2024-09-21T02:06:52.429068+0000","last_fullsized":"2024-09-21T02:06:52.429068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435335+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:06:52.435335+0000","last_peered":"2024-09-21T02:06:52.435335+0000","last_clean":"2024-09-21T02:06:52.435335+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:06:52.435335+0000","last_undegraded":"2024-09-21T02:06:52.435335+0000","last_fullsized":"2024-09-21T02:06:52.435335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438080+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:06:52.438080+0000","last_peered":"2024-09-21T02:06:52.438080+0000","last_clean":"2024-09-21T02:06:52.438080+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:06:52.438080+0000","last_undegraded":"2024-09-21T02:06:52.438080+0000","last_fullsized":"2024-09-21T02:06:52.438080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447599+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:06:52.447599+0000","last_peered":"2024-09-21T02:06:52.447599+0000","last_clean":"2024-09-21T02:06:52.447599+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:06:52.447599+0000","last_undegraded":"2024-09-21T02:06:52.447599+0000","last_fullsized":"2024-09-21T02:06:52.447599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446393+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:06:52.446393+0000","last_peered":"2024-09-21T02:06:52.446393+0000","last_clean":"2024-09-21T02:06:52.446393+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:06:52.446393+0000","last_undegraded":"2024-09-21T02:06:52.446393+0000","last_fullsized":"2024-09-21T02:06:52.446393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436021+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:06:52.436021+0000","last_peered":"2024-09-21T02:06:52.436021+0000","last_clean":"2024-09-21T02:06:52.436021+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:06:52.436021+0000","last_undegraded":"2024-09-21T02:06:52.436021+0000","last_fullsized":"2024-09-21T02:06:52.436021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436853+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:06:52.436853+0000","last_peered":"2024-09-21T02:06:52.436853+0000","last_clean":"2024-09-21T02:06:52.436853+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:06:52.436853+0000","last_undegraded":"2024-09-21T02:06:52.436853+0000","last_fullsized":"2024-09-21T02:06:52.436853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445987+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:06:52.445987+0000","last_peered":"2024-09-21T02:06:52.445987+0000","last_clean":"2024-09-21T02:06:52.445987+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:06:52.445987+0000","last_undegraded":"2024-09-21T02:06:52.445987+0000","last_fullsized":"2024-09-21T02:06:52.445987+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434828+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:06:52.434828+0000","last_peered":"2024-09-21T02:06:52.434828+0000","last_clean":"2024-09-21T02:06:52.434828+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:06:52.434828+0000","last_undegraded":"2024-09-21T02:06:52.434828+0000","last_fullsized":"2024-09-21T02:06:52.434828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447148+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:06:52.447148+0000","last_peered":"2024-09-21T02:06:52.447148+0000","last_clean":"2024-09-21T02:06:52.447148+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:06:52.447148+0000","last_undegraded":"2024-09-21T02:06:52.447148+0000","last_fullsized":"2024-09-21T02:06:52.447148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432764+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:06:52.432764+0000","last_peered":"2024-09-21T02:06:52.432764+0000","last_clean":"2024-09-21T02:06:52.432764+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:06:52.432764+0000","last_undegraded":"2024-09-21T02:06:52.432764+0000","last_fullsized":"2024-09-21T02:06:52.432764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428308+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:06:52.428308+0000","last_peered":"2024-09-21T02:06:52.428308+0000","last_clean":"2024-09-21T02:06:52.428308+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:06:52.428308+0000","last_undegraded":"2024-09-21T02:06:52.428308+0000","last_fullsized":"2024-09-21T02:06:52.428308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437194+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:06:52.437194+0000","last_peered":"2024-09-21T02:06:52.437194+0000","last_clean":"2024-09-21T02:06:52.437194+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:06:52.437194+0000","last_undegraded":"2024-09-21T02:06:52.437194+0000","last_fullsized":"2024-09-21T02:06:52.437194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430717+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:06:52.430717+0000","last_peered":"2024-09-21T02:06:52.430717+0000","last_clean":"2024-09-21T02:06:52.430717+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:06:52.430717+0000","last_undegraded":"2024-09-21T02:06:52.430717+0000","last_fullsized":"2024-09-21T02:06:52.430717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444213+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:06:52.444213+0000","last_peered":"2024-09-21T02:06:52.444213+0000","last_clean":"2024-09-21T02:06:52.444213+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:06:52.444213+0000","last_undegraded":"2024-09-21T02:06:52.444213+0000","last_fullsized":"2024-09-21T02:06:52.444213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432020+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:06:52.432020+0000","last_peered":"2024-09-21T02:06:52.432020+0000","last_clean":"2024-09-21T02:06:52.432020+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:06:52.432020+0000","last_undegraded":"2024-09-21T02:06:52.432020+0000","last_fullsized":"2024-09-21T02:06:52.432020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438660+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:06:52.438660+0000","last_peered":"2024-09-21T02:06:52.438660+0000","last_clean":"2024-09-21T02:06:52.438660+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:06:52.438660+0000","last_undegraded":"2024-09-21T02:06:52.438660+0000","last_fullsized":"2024-09-21T02:06:52.438660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436322+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:06:52.436322+0000","last_peered":"2024-09-21T02:06:52.436322+0000","last_clean":"2024-09-21T02:06:52.436322+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:06:52.436322+0000","last_undegraded":"2024-09-21T02:06:52.436322+0000","last_fullsized":"2024-09-21T02:06:52.436322+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435393+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:06:52.435393+0000","last_peered":"2024-09-21T02:06:52.435393+0000","last_clean":"2024-09-21T02:06:52.435393+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:06:52.435393+0000","last_undegraded":"2024-09-21T02:06:52.435393+0000","last_fullsized":"2024-09-21T02:06:52.435393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.426787+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:06:52.426787+0000","last_peered":"2024-09-21T02:06:52.426787+0000","last_clean":"2024-09-21T02:06:52.426787+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:06:52.426787+0000","last_undegraded":"2024-09-21T02:06:52.426787+0000","last_fullsized":"2024-09-21T02:06:52.426787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434645+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:06:52.434645+0000","last_peered":"2024-09-21T02:06:52.434645+0000","last_clean":"2024-09-21T02:06:52.434645+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:06:52.434645+0000","last_undegraded":"2024-09-21T02:06:52.434645+0000","last_fullsized":"2024-09-21T02:06:52.434645+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430839+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:06:52.430839+0000","last_peered":"2024-09-21T02:06:52.430839+0000","last_clean":"2024-09-21T02:06:52.430839+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:06:52.430839+0000","last_undegraded":"2024-09-21T02:06:52.430839+0000","last_fullsized":"2024-09-21T02:06:52.430839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.433464+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:06:52.433464+0000","last_peered":"2024-09-21T02:06:52.433464+0000","last_clean":"2024-09-21T02:06:52.433464+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:06:52.433464+0000","last_undegraded":"2024-09-21T02:06:52.433464+0000","last_fullsized":"2024-09-21T02:06:52.433464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427717+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:06:52.427717+0000","last_peered":"2024-09-21T02:06:52.427717+0000","last_clean":"2024-09-21T02:06:52.427717+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:06:52.427717+0000","last_undegraded":"2024-09-21T02:06:52.427717+0000","last_fullsized":"2024-09-21T02:06:52.427717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431461+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:06:52.431461+0000","last_peered":"2024-09-21T02:06:52.431461+0000","last_clean":"2024-09-21T02:06:52.431461+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:06:52.431461+0000","last_undegraded":"2024-09-21T02:06:52.431461+0000","last_fullsized":"2024-09-21T02:06:52.431461+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":471252,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":1143,"num_read_kb":666,"num_write":333,"num_write_kb":902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643072,"data_stored":625218,"data_compressed":17729,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710765,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":501553,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874289,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":1155211,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037815,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":504442,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234043,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":500612,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397568,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1098426,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561094,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":496382,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1093437,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":500949,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":495558,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378715,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":564513,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542240,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":564971,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705765,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":497838,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397621,"data_compressed":17729,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:07:11.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:07:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:12.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:12 smithi016 ceph-mon[31245]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:12.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:12 smithi123 ceph-mon[33453]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2161458454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac10de60 msgr2=0x7fbbac110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2161458454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac10de60 0x7fbbac110330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009ca0 tx=0x7fbba402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2161458454 shutdown_connections 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2161458454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac10de60 0x7fbbac110330 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2161458454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 0x7fbbac10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2161458454 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac1048b0 0x7fbbac104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2161458454 >> 172.21.15.16:0/2161458454 conn(0x7fbbac0796d0 msgr2=0x7fbbac079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2161458454 shutdown_connections 2024-09-21T02:07:12.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.586+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2161458454 wait complete. 2024-09-21T02:07:12.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 Processor -- start 2024-09-21T02:07:12.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 -- start start 2024-09-21T02:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 0x7fbbac19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac10de60 0x7fbbac19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac1139a0 con 0x7fbbac1048b0 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac113820 con 0x7fbbac105290 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbb0ecc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac113b20 con 0x7fbbac10de60 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.587+0000 7fbbaa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57710/0 (socket says 172.21.15.16:57710) 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 -- 172.21.15.16:0/2125568982 learned_addr learned my addr 172.21.15.16:0/2125568982 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaad9d640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac10de60 0x7fbbac19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbba9d9b640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 0x7fbbac19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 -- 172.21.15.16:0/2125568982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac10de60 msgr2=0x7fbbac19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac10de60 0x7fbbac19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 -- 172.21.15.16:0/2125568982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 msgr2=0x7fbbac19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 0x7fbbac19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 -- 172.21.15.16:0/2125568982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac1a0280 con 0x7fbbac1048b0 2024-09-21T02:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbaa59c640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00d950 tx=0x7fbb9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c017030 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbac1a0570 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.588+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb9c004020 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.589+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c0055d0 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.589+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbac1a6cd0 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.590+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbac06c190 con 0x7fbbac1048b0 2024-09-21T02:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.591+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fbb9c009a30 con 0x7fbbac1048b0 2024-09-21T02:07:12.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.592+0000 7fbba37fe640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 0x7fbb980b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:12.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.593+0000 7fbba9d9b640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 0x7fbb980b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:12.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.593+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fbb9c0e7b90 con 0x7fbbac1048b0 2024-09-21T02:07:12.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.593+0000 7fbba9d9b640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 0x7fbb980b8b00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac10fc60 tx=0x7fbb94005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:12.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.600+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbb9c0addd0 con 0x7fbbac1048b0 2024-09-21T02:07:12.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:12 smithi161 ceph-mon[34061]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:12.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.790+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbac101d90 con 0x7fbb980b65f0 2024-09-21T02:07:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.798+0000 7fbba37fe640 1 -- 172.21.15.16:0/2125568982 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340837 (secure 0 0 0) 0x7fbbac101d90 con 0x7fbb980b65f0 2024-09-21T02:07:12.802 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:12.804 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:07:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 msgr2=0x7fbb980b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 0x7fbb980b8b00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbbac10fc60 tx=0x7fbb94005f90 comp rx=0 tx=0).stop 2024-09-21T02:07:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 msgr2=0x7fbbac19a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00d950 tx=0x7fbb9c00de10 comp rx=0 tx=0).stop 2024-09-21T02:07:12.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 shutdown_connections 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb980b65f0 0x7fbb980b8b00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbac10de60 0x7fbbac19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac105290 0x7fbbac19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 --2- 172.21.15.16:0/2125568982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbac1048b0 0x7fbbac19a310 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 >> 172.21.15.16:0/2125568982 conn(0x7fbbac0796d0 msgr2=0x7fbbac100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 shutdown_connections 2024-09-21T02:07:12.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:12.802+0000 7fbbb0ecc640 1 -- 172.21.15.16:0/2125568982 wait complete. 2024-09-21T02:07:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:13 smithi016 ceph-mon[31245]: pgmap v354: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T02:07:13.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:13 smithi016 ceph-mon[31245]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:13.481 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-21T02:07:12.059991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1061854,"num_objects":103,"num_object_clones":0,"num_object_copies":311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":1240,"num_read_kb":741,"num_write":462,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":881,"ondisk_log_size":881,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394252,"kb_used_data":11724,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547572,"statfs":{"total":1151940427776,"available":1151536713728,"internally_reserved":0,"allocated":12005376,"data_stored":8375401,"data_compressed":484499,"data_compressed_allocated":651264,"data_compressed_original":3145728,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":396241,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":78,"num_write":73,"num_write_kb":746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003774"},"pg_stats":[{"pgid":"2.1f","version":"86'14","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:05.898888+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:07:05.898888+0000","last_peered":"2024-09-21T02:07:05.898888+0000","last_clean":"2024-09-21T02:07:05.898888+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:07:05.898888+0000","last_undegraded":"2024-09-21T02:07:05.898888+0000","last_fullsized":"2024-09-21T02:07:05.898888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435128+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:06:52.435128+0000","last_peered":"2024-09-21T02:06:52.435128+0000","last_clean":"2024-09-21T02:06:52.435128+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:06:52.435128+0000","last_undegraded":"2024-09-21T02:06:52.435128+0000","last_fullsized":"2024-09-21T02:06:52.435128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309551+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:05:54.309551+0000","last_peered":"2024-09-21T02:05:54.309551+0000","last_clean":"2024-09-21T02:05:54.309551+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:05:54.309551+0000","last_undegraded":"2024-09-21T02:05:54.309551+0000","last_fullsized":"2024-09-21T02:05:54.309551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694353+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:05:54.694353+0000","last_peered":"2024-09-21T02:05:54.694353+0000","last_clean":"2024-09-21T02:05:54.694353+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:05:54.694353+0000","last_undegraded":"2024-09-21T02:05:54.694353+0000","last_fullsized":"2024-09-21T02:05:54.694353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434557+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:06:52.434557+0000","last_peered":"2024-09-21T02:06:52.434557+0000","last_clean":"2024-09-21T02:06:52.434557+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:06:52.434557+0000","last_undegraded":"2024-09-21T02:06:52.434557+0000","last_fullsized":"2024-09-21T02:06:52.434557+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313872+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:05:54.313872+0000","last_peered":"2024-09-21T02:05:54.313872+0000","last_clean":"2024-09-21T02:05:54.313872+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:05:54.313872+0000","last_undegraded":"2024-09-21T02:05:54.313872+0000","last_fullsized":"2024-09-21T02:05:54.313872+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314180+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:54.314180+0000","last_peered":"2024-09-21T02:05:54.314180+0000","last_clean":"2024-09-21T02:05:54.314180+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:54.314180+0000","last_undegraded":"2024-09-21T02:05:54.314180+0000","last_fullsized":"2024-09-21T02:05:54.314180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447667+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:06:52.447667+0000","last_peered":"2024-09-21T02:06:52.447667+0000","last_clean":"2024-09-21T02:06:52.447667+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:06:52.447667+0000","last_undegraded":"2024-09-21T02:06:52.447667+0000","last_fullsized":"2024-09-21T02:06:52.447667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841186+0000","last_change":"2024-09-21T02:05:54.841083+0000","last_active":"2024-09-21T02:05:54.841186+0000","last_peered":"2024-09-21T02:05:54.841186+0000","last_clean":"2024-09-21T02:05:54.841186+0000","last_became_active":"2024-09-21T02:05:54.840645+0000","last_became_peered":"2024-09-21T02:05:54.840645+0000","last_unstale":"2024-09-21T02:05:54.841186+0000","last_undegraded":"2024-09-21T02:05:54.841186+0000","last_fullsized":"2024-09-21T02:05:54.841186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:33:35.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840304+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:54.840304+0000","last_peered":"2024-09-21T02:05:54.840304+0000","last_clean":"2024-09-21T02:05:54.840304+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:54.840304+0000","last_undegraded":"2024-09-21T02:05:54.840304+0000","last_fullsized":"2024-09-21T02:05:54.840304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445936+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:06:52.445936+0000","last_peered":"2024-09-21T02:06:52.445936+0000","last_clean":"2024-09-21T02:06:52.445936+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:06:52.445936+0000","last_undegraded":"2024-09-21T02:06:52.445936+0000","last_fullsized":"2024-09-21T02:06:52.445936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313927+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:05:54.313927+0000","last_peered":"2024-09-21T02:05:54.313927+0000","last_clean":"2024-09-21T02:05:54.313927+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:05:54.313927+0000","last_undegraded":"2024-09-21T02:05:54.313927+0000","last_fullsized":"2024-09-21T02:05:54.313927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445821+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:06:52.445821+0000","last_peered":"2024-09-21T02:06:52.445821+0000","last_clean":"2024-09-21T02:06:52.445821+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:06:52.445821+0000","last_undegraded":"2024-09-21T02:06:52.445821+0000","last_fullsized":"2024-09-21T02:06:52.445821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":56,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839948+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:54.839948+0000","last_peered":"2024-09-21T02:05:54.839948+0000","last_clean":"2024-09-21T02:05:54.839948+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:54.839948+0000","last_undegraded":"2024-09-21T02:05:54.839948+0000","last_fullsized":"2024-09-21T02:05:54.839948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314223+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:05:54.314223+0000","last_peered":"2024-09-21T02:05:54.314223+0000","last_clean":"2024-09-21T02:05:54.314223+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:05:54.314223+0000","last_undegraded":"2024-09-21T02:05:54.314223+0000","last_fullsized":"2024-09-21T02:05:54.314223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432917+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:06:52.432917+0000","last_peered":"2024-09-21T02:06:52.432917+0000","last_clean":"2024-09-21T02:06:52.432917+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:06:52.432917+0000","last_undegraded":"2024-09-21T02:06:52.432917+0000","last_fullsized":"2024-09-21T02:06:52.432917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:41.387929+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:06:41.387929+0000","last_peered":"2024-09-21T02:06:41.387929+0000","last_clean":"2024-09-21T02:06:41.387929+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:06:41.387929+0000","last_undegraded":"2024-09-21T02:06:41.387929+0000","last_fullsized":"2024-09-21T02:06:41.387929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314252+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:05:54.314252+0000","last_peered":"2024-09-21T02:05:54.314252+0000","last_clean":"2024-09-21T02:05:54.314252+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:05:54.314252+0000","last_undegraded":"2024-09-21T02:05:54.314252+0000","last_fullsized":"2024-09-21T02:05:54.314252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428361+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.428361+0000","last_peered":"2024-09-21T02:06:52.428361+0000","last_clean":"2024-09-21T02:06:52.428361+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:06:52.428361+0000","last_undegraded":"2024-09-21T02:06:52.428361+0000","last_fullsized":"2024-09-21T02:06:52.428361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693914+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:54.693914+0000","last_peered":"2024-09-21T02:05:54.693914+0000","last_clean":"2024-09-21T02:05:54.693914+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:54.693914+0000","last_undegraded":"2024-09-21T02:05:54.693914+0000","last_fullsized":"2024-09-21T02:05:54.693914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.754062+0000","last_change":"2024-09-21T02:05:54.753929+0000","last_active":"2024-09-21T02:05:54.754062+0000","last_peered":"2024-09-21T02:05:54.754062+0000","last_clean":"2024-09-21T02:05:54.754062+0000","last_became_active":"2024-09-21T02:05:54.753431+0000","last_became_peered":"2024-09-21T02:05:54.753431+0000","last_unstale":"2024-09-21T02:05:54.754062+0000","last_undegraded":"2024-09-21T02:05:54.754062+0000","last_fullsized":"2024-09-21T02:05:54.754062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:20:41.311972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.391580+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:06:52.391580+0000","last_peered":"2024-09-21T02:06:52.391580+0000","last_clean":"2024-09-21T02:06:52.391580+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:06:52.391580+0000","last_undegraded":"2024-09-21T02:06:52.391580+0000","last_fullsized":"2024-09-21T02:06:52.391580+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839098+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:54.839098+0000","last_peered":"2024-09-21T02:05:54.839098+0000","last_clean":"2024-09-21T02:05:54.839098+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:54.839098+0000","last_undegraded":"2024-09-21T02:05:54.839098+0000","last_fullsized":"2024-09-21T02:05:54.839098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314296+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:05:54.314296+0000","last_peered":"2024-09-21T02:05:54.314296+0000","last_clean":"2024-09-21T02:05:54.314296+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:05:54.314296+0000","last_undegraded":"2024-09-21T02:05:54.314296+0000","last_fullsized":"2024-09-21T02:05:54.314296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432105+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:06:52.432105+0000","last_peered":"2024-09-21T02:06:52.432105+0000","last_clean":"2024-09-21T02:06:52.432105+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:06:52.432105+0000","last_undegraded":"2024-09-21T02:06:52.432105+0000","last_fullsized":"2024-09-21T02:06:52.432105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314615+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:54.314615+0000","last_peered":"2024-09-21T02:05:54.314615+0000","last_clean":"2024-09-21T02:05:54.314615+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:54.314615+0000","last_undegraded":"2024-09-21T02:05:54.314615+0000","last_fullsized":"2024-09-21T02:05:54.314615+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309689+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:05:54.309689+0000","last_peered":"2024-09-21T02:05:54.309689+0000","last_clean":"2024-09-21T02:05:54.309689+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:05:54.309689+0000","last_undegraded":"2024-09-21T02:05:54.309689+0000","last_fullsized":"2024-09-21T02:05:54.309689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437884+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:06:52.437884+0000","last_peered":"2024-09-21T02:06:52.437884+0000","last_clean":"2024-09-21T02:06:52.437884+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:06:52.437884+0000","last_undegraded":"2024-09-21T02:06:52.437884+0000","last_fullsized":"2024-09-21T02:06:52.437884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'18","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.675186+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:06:52.675186+0000","last_peered":"2024-09-21T02:06:52.675186+0000","last_clean":"2024-09-21T02:06:52.675186+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:06:52.675186+0000","last_undegraded":"2024-09-21T02:06:52.675186+0000","last_fullsized":"2024-09-21T02:06:52.675186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693575+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:05:54.693575+0000","last_peered":"2024-09-21T02:05:54.693575+0000","last_clean":"2024-09-21T02:05:54.693575+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:05:54.693575+0000","last_undegraded":"2024-09-21T02:05:54.693575+0000","last_fullsized":"2024-09-21T02:05:54.693575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430290+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:06:52.430290+0000","last_peered":"2024-09-21T02:06:52.430290+0000","last_clean":"2024-09-21T02:06:52.430290+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:06:52.430290+0000","last_undegraded":"2024-09-21T02:06:52.430290+0000","last_fullsized":"2024-09-21T02:06:52.430290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314156+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:54.314156+0000","last_peered":"2024-09-21T02:05:54.314156+0000","last_clean":"2024-09-21T02:05:54.314156+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:54.314156+0000","last_undegraded":"2024-09-21T02:05:54.314156+0000","last_fullsized":"2024-09-21T02:05:54.314156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314000+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:05:54.314000+0000","last_peered":"2024-09-21T02:05:54.314000+0000","last_clean":"2024-09-21T02:05:54.314000+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:05:54.314000+0000","last_undegraded":"2024-09-21T02:05:54.314000+0000","last_fullsized":"2024-09-21T02:05:54.314000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431581+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.431581+0000","last_peered":"2024-09-21T02:06:52.431581+0000","last_clean":"2024-09-21T02:06:52.431581+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:06:52.431581+0000","last_undegraded":"2024-09-21T02:06:52.431581+0000","last_fullsized":"2024-09-21T02:06:52.431581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796090+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:05:54.796090+0000","last_peered":"2024-09-21T02:05:54.796090+0000","last_clean":"2024-09-21T02:05:54.796090+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:05:54.796090+0000","last_undegraded":"2024-09-21T02:05:54.796090+0000","last_fullsized":"2024-09-21T02:05:54.796090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839552+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:05:54.839552+0000","last_peered":"2024-09-21T02:05:54.839552+0000","last_clean":"2024-09-21T02:05:54.839552+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:05:54.839552+0000","last_undegraded":"2024-09-21T02:05:54.839552+0000","last_fullsized":"2024-09-21T02:05:54.839552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427901+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:06:52.427901+0000","last_peered":"2024-09-21T02:06:52.427901+0000","last_clean":"2024-09-21T02:06:52.427901+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:06:52.427901+0000","last_undegraded":"2024-09-21T02:06:52.427901+0000","last_fullsized":"2024-09-21T02:06:52.427901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'8","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.905124+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:07:08.905124+0000","last_peered":"2024-09-21T02:07:08.905124+0000","last_clean":"2024-09-21T02:07:08.905124+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:07:08.905124+0000","last_undegraded":"2024-09-21T02:07:08.905124+0000","last_fullsized":"2024-09-21T02:07:08.905124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308834+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:05:54.308834+0000","last_peered":"2024-09-21T02:05:54.308834+0000","last_clean":"2024-09-21T02:05:54.308834+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:05:54.308834+0000","last_undegraded":"2024-09-21T02:05:54.308834+0000","last_fullsized":"2024-09-21T02:05:54.308834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431202+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:06:52.431202+0000","last_peered":"2024-09-21T02:06:52.431202+0000","last_clean":"2024-09-21T02:06:52.431202+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:06:52.431202+0000","last_undegraded":"2024-09-21T02:06:52.431202+0000","last_fullsized":"2024-09-21T02:06:52.431202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":157,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.904571+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:07:08.904571+0000","last_peered":"2024-09-21T02:07:08.904571+0000","last_clean":"2024-09-21T02:07:08.904571+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:07:08.904571+0000","last_undegraded":"2024-09-21T02:07:08.904571+0000","last_fullsized":"2024-09-21T02:07:08.904571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309231+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:05:54.309231+0000","last_peered":"2024-09-21T02:05:54.309231+0000","last_clean":"2024-09-21T02:05:54.309231+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:05:54.309231+0000","last_undegraded":"2024-09-21T02:05:54.309231+0000","last_fullsized":"2024-09-21T02:05:54.309231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431015+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:06:52.431015+0000","last_peered":"2024-09-21T02:06:52.431015+0000","last_clean":"2024-09-21T02:06:52.431015+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:06:52.431015+0000","last_undegraded":"2024-09-21T02:06:52.431015+0000","last_fullsized":"2024-09-21T02:06:52.431015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799870+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:54.799870+0000","last_peered":"2024-09-21T02:05:54.799870+0000","last_clean":"2024-09-21T02:05:54.799870+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:54.799870+0000","last_undegraded":"2024-09-21T02:05:54.799870+0000","last_fullsized":"2024-09-21T02:05:54.799870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752633+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:05:54.752633+0000","last_peered":"2024-09-21T02:05:54.752633+0000","last_clean":"2024-09-21T02:05:54.752633+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:05:54.752633+0000","last_undegraded":"2024-09-21T02:05:54.752633+0000","last_fullsized":"2024-09-21T02:05:54.752633+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436058+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:06:52.436058+0000","last_peered":"2024-09-21T02:06:52.436058+0000","last_clean":"2024-09-21T02:06:52.436058+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:06:52.436058+0000","last_undegraded":"2024-09-21T02:06:52.436058+0000","last_fullsized":"2024-09-21T02:06:52.436058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314486+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:54.314486+0000","last_peered":"2024-09-21T02:05:54.314486+0000","last_clean":"2024-09-21T02:05:54.314486+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:54.314486+0000","last_undegraded":"2024-09-21T02:05:54.314486+0000","last_fullsized":"2024-09-21T02:05:54.314486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799508+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:05:54.799508+0000","last_peered":"2024-09-21T02:05:54.799508+0000","last_clean":"2024-09-21T02:05:54.799508+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:05:54.799508+0000","last_undegraded":"2024-09-21T02:05:54.799508+0000","last_fullsized":"2024-09-21T02:05:54.799508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799546+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:05:54.799546+0000","last_peered":"2024-09-21T02:05:54.799546+0000","last_clean":"2024-09-21T02:05:54.799546+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:05:54.799546+0000","last_undegraded":"2024-09-21T02:05:54.799546+0000","last_fullsized":"2024-09-21T02:05:54.799546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444281+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:06:52.444281+0000","last_peered":"2024-09-21T02:06:52.444281+0000","last_clean":"2024-09-21T02:06:52.444281+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:06:52.444281+0000","last_undegraded":"2024-09-21T02:06:52.444281+0000","last_fullsized":"2024-09-21T02:06:52.444281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'8","reported_seq":164,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:09.920672+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:07:09.920672+0000","last_peered":"2024-09-21T02:07:09.920672+0000","last_clean":"2024-09-21T02:07:09.920672+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:07:09.920672+0000","last_undegraded":"2024-09-21T02:07:09.920672+0000","last_fullsized":"2024-09-21T02:07:09.920672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838953+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:05:54.838953+0000","last_peered":"2024-09-21T02:05:54.838953+0000","last_clean":"2024-09-21T02:05:54.838953+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:05:54.838953+0000","last_undegraded":"2024-09-21T02:05:54.838953+0000","last_fullsized":"2024-09-21T02:05:54.838953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435516+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:06:52.435516+0000","last_peered":"2024-09-21T02:06:52.435516+0000","last_clean":"2024-09-21T02:06:52.435516+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:06:52.435516+0000","last_undegraded":"2024-09-21T02:06:52.435516+0000","last_fullsized":"2024-09-21T02:06:52.435516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'54","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.723869+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:07:08.723869+0000","last_peered":"2024-09-21T02:07:08.723869+0000","last_clean":"2024-09-21T02:07:08.723869+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:07:08.723869+0000","last_undegraded":"2024-09-21T02:07:08.723869+0000","last_fullsized":"2024-09-21T02:07:08.723869+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":63,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838590+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:05:54.838590+0000","last_peered":"2024-09-21T02:05:54.838590+0000","last_clean":"2024-09-21T02:05:54.838590+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:05:54.838590+0000","last_undegraded":"2024-09-21T02:05:54.838590+0000","last_fullsized":"2024-09-21T02:05:54.838590+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.443855+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:06:52.443855+0000","last_peered":"2024-09-21T02:06:52.443855+0000","last_clean":"2024-09-21T02:06:52.443855+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:06:52.443855+0000","last_undegraded":"2024-09-21T02:06:52.443855+0000","last_fullsized":"2024-09-21T02:06:52.443855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309244+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:54.309244+0000","last_peered":"2024-09-21T02:05:54.309244+0000","last_clean":"2024-09-21T02:05:54.309244+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:54.309244+0000","last_undegraded":"2024-09-21T02:05:54.309244+0000","last_fullsized":"2024-09-21T02:05:54.309244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751778+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:05:54.751778+0000","last_peered":"2024-09-21T02:05:54.751778+0000","last_clean":"2024-09-21T02:05:54.751778+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:05:54.751778+0000","last_undegraded":"2024-09-21T02:05:54.751778+0000","last_fullsized":"2024-09-21T02:05:54.751778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436869+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:06:52.436869+0000","last_peered":"2024-09-21T02:06:52.436869+0000","last_clean":"2024-09-21T02:06:52.436869+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:06:52.436869+0000","last_undegraded":"2024-09-21T02:06:52.436869+0000","last_fullsized":"2024-09-21T02:06:52.436869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'3","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:45.953185+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:06:45.953185+0000","last_peered":"2024-09-21T02:06:45.953185+0000","last_clean":"2024-09-21T02:06:45.953185+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:06:45.953185+0000","last_undegraded":"2024-09-21T02:06:45.953185+0000","last_fullsized":"2024-09-21T02:06:45.953185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751987+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:05:54.751987+0000","last_peered":"2024-09-21T02:05:54.751987+0000","last_clean":"2024-09-21T02:05:54.751987+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:05:54.751987+0000","last_undegraded":"2024-09-21T02:05:54.751987+0000","last_fullsized":"2024-09-21T02:05:54.751987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437619+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:06:52.437619+0000","last_peered":"2024-09-21T02:06:52.437619+0000","last_clean":"2024-09-21T02:06:52.437619+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:06:52.437619+0000","last_undegraded":"2024-09-21T02:06:52.437619+0000","last_fullsized":"2024-09-21T02:06:52.437619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839608+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:54.839608+0000","last_peered":"2024-09-21T02:05:54.839608+0000","last_clean":"2024-09-21T02:05:54.839608+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:54.839608+0000","last_undegraded":"2024-09-21T02:05:54.839608+0000","last_fullsized":"2024-09-21T02:05:54.839608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309217+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:05:54.309217+0000","last_peered":"2024-09-21T02:05:54.309217+0000","last_clean":"2024-09-21T02:05:54.309217+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:05:54.309217+0000","last_undegraded":"2024-09-21T02:05:54.309217+0000","last_fullsized":"2024-09-21T02:05:54.309217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445290+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:06:52.445290+0000","last_peered":"2024-09-21T02:06:52.445290+0000","last_clean":"2024-09-21T02:06:52.445290+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:06:52.445290+0000","last_undegraded":"2024-09-21T02:06:52.445290+0000","last_fullsized":"2024-09-21T02:06:52.445290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'8","reported_seq":161,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:06.902308+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:07:06.902308+0000","last_peered":"2024-09-21T02:07:06.902308+0000","last_clean":"2024-09-21T02:07:06.902308+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:07:06.902308+0000","last_undegraded":"2024-09-21T02:07:06.902308+0000","last_fullsized":"2024-09-21T02:07:06.902308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796399+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:05:54.796399+0000","last_peered":"2024-09-21T02:05:54.796399+0000","last_clean":"2024-09-21T02:05:54.796399+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:05:54.796399+0000","last_undegraded":"2024-09-21T02:05:54.796399+0000","last_fullsized":"2024-09-21T02:05:54.796399+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446922+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:06:52.446922+0000","last_peered":"2024-09-21T02:06:52.446922+0000","last_clean":"2024-09-21T02:06:52.446922+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:06:52.446922+0000","last_undegraded":"2024-09-21T02:06:52.446922+0000","last_fullsized":"2024-09-21T02:06:52.446922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752428+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:54.752428+0000","last_peered":"2024-09-21T02:05:54.752428+0000","last_clean":"2024-09-21T02:05:54.752428+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:54.752428+0000","last_undegraded":"2024-09-21T02:05:54.752428+0000","last_fullsized":"2024-09-21T02:05:54.752428+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314337+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:05:54.314337+0000","last_peered":"2024-09-21T02:05:54.314337+0000","last_clean":"2024-09-21T02:05:54.314337+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:05:54.314337+0000","last_undegraded":"2024-09-21T02:05:54.314337+0000","last_fullsized":"2024-09-21T02:05:54.314337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446213+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:06:52.446213+0000","last_peered":"2024-09-21T02:06:52.446213+0000","last_clean":"2024-09-21T02:06:52.446213+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:06:52.446213+0000","last_undegraded":"2024-09-21T02:06:52.446213+0000","last_fullsized":"2024-09-21T02:06:52.446213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309963+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:54.309963+0000","last_peered":"2024-09-21T02:05:54.309963+0000","last_clean":"2024-09-21T02:05:54.309963+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:54.309963+0000","last_undegraded":"2024-09-21T02:05:54.309963+0000","last_fullsized":"2024-09-21T02:05:54.309963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752780+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:05:54.752780+0000","last_peered":"2024-09-21T02:05:54.752780+0000","last_clean":"2024-09-21T02:05:54.752780+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:05:54.752780+0000","last_undegraded":"2024-09-21T02:05:54.752780+0000","last_fullsized":"2024-09-21T02:05:54.752780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.424275+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:06:52.424275+0000","last_peered":"2024-09-21T02:06:52.424275+0000","last_clean":"2024-09-21T02:06:52.424275+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:06:52.424275+0000","last_undegraded":"2024-09-21T02:06:52.424275+0000","last_fullsized":"2024-09-21T02:06:52.424275+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'38","reported_seq":305,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.920653+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:07:07.920653+0000","last_peered":"2024-09-21T02:07:07.920653+0000","last_clean":"2024-09-21T02:07:07.920653+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:07:07.920653+0000","last_undegraded":"2024-09-21T02:07:07.920653+0000","last_fullsized":"2024-09-21T02:07:07.920653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66954,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":227,"num_read_kb":179,"num_write":40,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795725+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:05:54.795725+0000","last_peered":"2024-09-21T02:05:54.795725+0000","last_clean":"2024-09-21T02:05:54.795725+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:05:54.795725+0000","last_undegraded":"2024-09-21T02:05:54.795725+0000","last_fullsized":"2024-09-21T02:05:54.795725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444121+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:06:52.444121+0000","last_peered":"2024-09-21T02:06:52.444121+0000","last_clean":"2024-09-21T02:06:52.444121+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:06:52.444121+0000","last_undegraded":"2024-09-21T02:06:52.444121+0000","last_fullsized":"2024-09-21T02:06:52.444121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752935+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:54.752935+0000","last_peered":"2024-09-21T02:05:54.752935+0000","last_clean":"2024-09-21T02:05:54.752935+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:54.752935+0000","last_undegraded":"2024-09-21T02:05:54.752935+0000","last_fullsized":"2024-09-21T02:05:54.752935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314193+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:05:54.314193+0000","last_peered":"2024-09-21T02:05:54.314193+0000","last_clean":"2024-09-21T02:05:54.314193+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:05:54.314193+0000","last_undegraded":"2024-09-21T02:05:54.314193+0000","last_fullsized":"2024-09-21T02:05:54.314193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437003+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:06:52.437003+0000","last_peered":"2024-09-21T02:06:52.437003+0000","last_clean":"2024-09-21T02:06:52.437003+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:06:52.437003+0000","last_undegraded":"2024-09-21T02:06:52.437003+0000","last_fullsized":"2024-09-21T02:06:52.437003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799684+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:54.799684+0000","last_peered":"2024-09-21T02:05:54.799684+0000","last_clean":"2024-09-21T02:05:54.799684+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:54.799684+0000","last_undegraded":"2024-09-21T02:05:54.799684+0000","last_fullsized":"2024-09-21T02:05:54.799684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839390+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:05:54.839390+0000","last_peered":"2024-09-21T02:05:54.839390+0000","last_clean":"2024-09-21T02:05:54.839390+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:05:54.839390+0000","last_undegraded":"2024-09-21T02:05:54.839390+0000","last_fullsized":"2024-09-21T02:05:54.839390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447850+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:06:52.447850+0000","last_peered":"2024-09-21T02:06:52.447850+0000","last_clean":"2024-09-21T02:06:52.447850+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:06:52.447850+0000","last_undegraded":"2024-09-21T02:06:52.447850+0000","last_fullsized":"2024-09-21T02:06:52.447850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'6","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.903683+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:07:07.903683+0000","last_peered":"2024-09-21T02:07:07.903683+0000","last_clean":"2024-09-21T02:07:07.903683+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:07:07.903683+0000","last_undegraded":"2024-09-21T02:07:07.903683+0000","last_fullsized":"2024-09-21T02:07:07.903683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693641+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:05:54.693641+0000","last_peered":"2024-09-21T02:05:54.693641+0000","last_clean":"2024-09-21T02:05:54.693641+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:05:54.693641+0000","last_undegraded":"2024-09-21T02:05:54.693641+0000","last_fullsized":"2024-09-21T02:05:54.693641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838683+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:05:54.838683+0000","last_peered":"2024-09-21T02:05:54.838683+0000","last_clean":"2024-09-21T02:05:54.838683+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:05:54.838683+0000","last_undegraded":"2024-09-21T02:05:54.838683+0000","last_fullsized":"2024-09-21T02:05:54.838683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429092+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:06:52.429092+0000","last_peered":"2024-09-21T02:06:52.429092+0000","last_clean":"2024-09-21T02:06:52.429092+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:06:52.429092+0000","last_undegraded":"2024-09-21T02:06:52.429092+0000","last_fullsized":"2024-09-21T02:06:52.429092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752288+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:54.752288+0000","last_peered":"2024-09-21T02:05:54.752288+0000","last_clean":"2024-09-21T02:05:54.752288+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:54.752288+0000","last_undegraded":"2024-09-21T02:05:54.752288+0000","last_fullsized":"2024-09-21T02:05:54.752288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.842081+0000","last_change":"2024-09-21T02:05:54.841426+0000","last_active":"2024-09-21T02:05:54.842081+0000","last_peered":"2024-09-21T02:05:54.842081+0000","last_clean":"2024-09-21T02:05:54.842081+0000","last_became_active":"2024-09-21T02:05:54.840855+0000","last_became_peered":"2024-09-21T02:05:54.840855+0000","last_unstale":"2024-09-21T02:05:54.842081+0000","last_undegraded":"2024-09-21T02:05:54.842081+0000","last_fullsized":"2024-09-21T02:05:54.842081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:40.637427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438022+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:06:52.438022+0000","last_peered":"2024-09-21T02:06:52.438022+0000","last_clean":"2024-09-21T02:06:52.438022+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:06:52.438022+0000","last_undegraded":"2024-09-21T02:06:52.438022+0000","last_fullsized":"2024-09-21T02:06:52.438022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308608+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:54.308608+0000","last_peered":"2024-09-21T02:05:54.308608+0000","last_clean":"2024-09-21T02:05:54.308608+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:54.308608+0000","last_undegraded":"2024-09-21T02:05:54.308608+0000","last_fullsized":"2024-09-21T02:05:54.308608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309526+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:54.309526+0000","last_peered":"2024-09-21T02:05:54.309526+0000","last_clean":"2024-09-21T02:05:54.309526+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:54.309526+0000","last_undegraded":"2024-09-21T02:05:54.309526+0000","last_fullsized":"2024-09-21T02:05:54.309526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437429+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:06:52.437429+0000","last_peered":"2024-09-21T02:06:52.437429+0000","last_clean":"2024-09-21T02:06:52.437429+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:06:52.437429+0000","last_undegraded":"2024-09-21T02:06:52.437429+0000","last_fullsized":"2024-09-21T02:06:52.437429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309835+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:05:54.309835+0000","last_peered":"2024-09-21T02:05:54.309835+0000","last_clean":"2024-09-21T02:05:54.309835+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:05:54.309835+0000","last_undegraded":"2024-09-21T02:05:54.309835+0000","last_fullsized":"2024-09-21T02:05:54.309835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'8","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:07.904253+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:07:07.904253+0000","last_peered":"2024-09-21T02:07:07.904253+0000","last_clean":"2024-09-21T02:07:07.904253+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:07:07.904253+0000","last_undegraded":"2024-09-21T02:07:07.904253+0000","last_fullsized":"2024-09-21T02:07:07.904253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.420899+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:06:52.420899+0000","last_peered":"2024-09-21T02:06:52.420899+0000","last_clean":"2024-09-21T02:06:52.420899+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:06:52.420899+0000","last_undegraded":"2024-09-21T02:06:52.420899+0000","last_fullsized":"2024-09-21T02:06:52.420899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309917+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:05:54.309917+0000","last_peered":"2024-09-21T02:05:54.309917+0000","last_clean":"2024-09-21T02:05:54.309917+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:05:54.309917+0000","last_undegraded":"2024-09-21T02:05:54.309917+0000","last_fullsized":"2024-09-21T02:05:54.309917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437860+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:06:52.437860+0000","last_peered":"2024-09-21T02:06:52.437860+0000","last_clean":"2024-09-21T02:06:52.437860+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:06:52.437860+0000","last_undegraded":"2024-09-21T02:06:52.437860+0000","last_fullsized":"2024-09-21T02:06:52.437860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435988+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:06:52.435988+0000","last_peered":"2024-09-21T02:06:52.435988+0000","last_clean":"2024-09-21T02:06:52.435988+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:06:52.435988+0000","last_undegraded":"2024-09-21T02:06:52.435988+0000","last_fullsized":"2024-09-21T02:06:52.435988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427485+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:06:52.427485+0000","last_peered":"2024-09-21T02:06:52.427485+0000","last_clean":"2024-09-21T02:06:52.427485+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:06:52.427485+0000","last_undegraded":"2024-09-21T02:06:52.427485+0000","last_fullsized":"2024-09-21T02:06:52.427485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435191+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:06:52.435191+0000","last_peered":"2024-09-21T02:06:52.435191+0000","last_clean":"2024-09-21T02:06:52.435191+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:06:52.435191+0000","last_undegraded":"2024-09-21T02:06:52.435191+0000","last_fullsized":"2024-09-21T02:06:52.435191+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431472+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:06:52.431472+0000","last_peered":"2024-09-21T02:06:52.431472+0000","last_clean":"2024-09-21T02:06:52.431472+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:06:52.431472+0000","last_undegraded":"2024-09-21T02:06:52.431472+0000","last_fullsized":"2024-09-21T02:06:52.431472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436873+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:06:52.436873+0000","last_peered":"2024-09-21T02:06:52.436873+0000","last_clean":"2024-09-21T02:06:52.436873+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:06:52.436873+0000","last_undegraded":"2024-09-21T02:06:52.436873+0000","last_fullsized":"2024-09-21T02:06:52.436873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.421562+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:06:52.421562+0000","last_peered":"2024-09-21T02:06:52.421562+0000","last_clean":"2024-09-21T02:06:52.421562+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:06:52.421562+0000","last_undegraded":"2024-09-21T02:06:52.421562+0000","last_fullsized":"2024-09-21T02:06:52.421562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429068+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:06:52.429068+0000","last_peered":"2024-09-21T02:06:52.429068+0000","last_clean":"2024-09-21T02:06:52.429068+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:06:52.429068+0000","last_undegraded":"2024-09-21T02:06:52.429068+0000","last_fullsized":"2024-09-21T02:06:52.429068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435335+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:06:52.435335+0000","last_peered":"2024-09-21T02:06:52.435335+0000","last_clean":"2024-09-21T02:06:52.435335+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:06:52.435335+0000","last_undegraded":"2024-09-21T02:06:52.435335+0000","last_fullsized":"2024-09-21T02:06:52.435335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438080+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:06:52.438080+0000","last_peered":"2024-09-21T02:06:52.438080+0000","last_clean":"2024-09-21T02:06:52.438080+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:06:52.438080+0000","last_undegraded":"2024-09-21T02:06:52.438080+0000","last_fullsized":"2024-09-21T02:06:52.438080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447599+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:06:52.447599+0000","last_peered":"2024-09-21T02:06:52.447599+0000","last_clean":"2024-09-21T02:06:52.447599+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:06:52.447599+0000","last_undegraded":"2024-09-21T02:06:52.447599+0000","last_fullsized":"2024-09-21T02:06:52.447599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446393+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:06:52.446393+0000","last_peered":"2024-09-21T02:06:52.446393+0000","last_clean":"2024-09-21T02:06:52.446393+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:06:52.446393+0000","last_undegraded":"2024-09-21T02:06:52.446393+0000","last_fullsized":"2024-09-21T02:06:52.446393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436021+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:06:52.436021+0000","last_peered":"2024-09-21T02:06:52.436021+0000","last_clean":"2024-09-21T02:06:52.436021+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:06:52.436021+0000","last_undegraded":"2024-09-21T02:06:52.436021+0000","last_fullsized":"2024-09-21T02:06:52.436021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436853+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:06:52.436853+0000","last_peered":"2024-09-21T02:06:52.436853+0000","last_clean":"2024-09-21T02:06:52.436853+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:06:52.436853+0000","last_undegraded":"2024-09-21T02:06:52.436853+0000","last_fullsized":"2024-09-21T02:06:52.436853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445987+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:06:52.445987+0000","last_peered":"2024-09-21T02:06:52.445987+0000","last_clean":"2024-09-21T02:06:52.445987+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:06:52.445987+0000","last_undegraded":"2024-09-21T02:06:52.445987+0000","last_fullsized":"2024-09-21T02:06:52.445987+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434828+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:06:52.434828+0000","last_peered":"2024-09-21T02:06:52.434828+0000","last_clean":"2024-09-21T02:06:52.434828+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:06:52.434828+0000","last_undegraded":"2024-09-21T02:06:52.434828+0000","last_fullsized":"2024-09-21T02:06:52.434828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447148+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:06:52.447148+0000","last_peered":"2024-09-21T02:06:52.447148+0000","last_clean":"2024-09-21T02:06:52.447148+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:06:52.447148+0000","last_undegraded":"2024-09-21T02:06:52.447148+0000","last_fullsized":"2024-09-21T02:06:52.447148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432764+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:06:52.432764+0000","last_peered":"2024-09-21T02:06:52.432764+0000","last_clean":"2024-09-21T02:06:52.432764+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:06:52.432764+0000","last_undegraded":"2024-09-21T02:06:52.432764+0000","last_fullsized":"2024-09-21T02:06:52.432764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428308+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:06:52.428308+0000","last_peered":"2024-09-21T02:06:52.428308+0000","last_clean":"2024-09-21T02:06:52.428308+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:06:52.428308+0000","last_undegraded":"2024-09-21T02:06:52.428308+0000","last_fullsized":"2024-09-21T02:06:52.428308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437194+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:06:52.437194+0000","last_peered":"2024-09-21T02:06:52.437194+0000","last_clean":"2024-09-21T02:06:52.437194+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:06:52.437194+0000","last_undegraded":"2024-09-21T02:06:52.437194+0000","last_fullsized":"2024-09-21T02:06:52.437194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430717+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:06:52.430717+0000","last_peered":"2024-09-21T02:06:52.430717+0000","last_clean":"2024-09-21T02:06:52.430717+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:06:52.430717+0000","last_undegraded":"2024-09-21T02:06:52.430717+0000","last_fullsized":"2024-09-21T02:06:52.430717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444213+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:06:52.444213+0000","last_peered":"2024-09-21T02:06:52.444213+0000","last_clean":"2024-09-21T02:06:52.444213+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:06:52.444213+0000","last_undegraded":"2024-09-21T02:06:52.444213+0000","last_fullsized":"2024-09-21T02:06:52.444213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432020+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:06:52.432020+0000","last_peered":"2024-09-21T02:06:52.432020+0000","last_clean":"2024-09-21T02:06:52.432020+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:06:52.432020+0000","last_undegraded":"2024-09-21T02:06:52.432020+0000","last_fullsized":"2024-09-21T02:06:52.432020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438660+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:06:52.438660+0000","last_peered":"2024-09-21T02:06:52.438660+0000","last_clean":"2024-09-21T02:06:52.438660+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:06:52.438660+0000","last_undegraded":"2024-09-21T02:06:52.438660+0000","last_fullsized":"2024-09-21T02:06:52.438660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436322+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:06:52.436322+0000","last_peered":"2024-09-21T02:06:52.436322+0000","last_clean":"2024-09-21T02:06:52.436322+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:06:52.436322+0000","last_undegraded":"2024-09-21T02:06:52.436322+0000","last_fullsized":"2024-09-21T02:06:52.436322+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435393+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:06:52.435393+0000","last_peered":"2024-09-21T02:06:52.435393+0000","last_clean":"2024-09-21T02:06:52.435393+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:06:52.435393+0000","last_undegraded":"2024-09-21T02:06:52.435393+0000","last_fullsized":"2024-09-21T02:06:52.435393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.426787+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:06:52.426787+0000","last_peered":"2024-09-21T02:06:52.426787+0000","last_clean":"2024-09-21T02:06:52.426787+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:06:52.426787+0000","last_undegraded":"2024-09-21T02:06:52.426787+0000","last_fullsized":"2024-09-21T02:06:52.426787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434645+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:06:52.434645+0000","last_peered":"2024-09-21T02:06:52.434645+0000","last_clean":"2024-09-21T02:06:52.434645+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:06:52.434645+0000","last_undegraded":"2024-09-21T02:06:52.434645+0000","last_fullsized":"2024-09-21T02:06:52.434645+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430839+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:06:52.430839+0000","last_peered":"2024-09-21T02:06:52.430839+0000","last_clean":"2024-09-21T02:06:52.430839+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:06:52.430839+0000","last_undegraded":"2024-09-21T02:06:52.430839+0000","last_fullsized":"2024-09-21T02:06:52.430839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.433464+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:06:52.433464+0000","last_peered":"2024-09-21T02:06:52.433464+0000","last_clean":"2024-09-21T02:06:52.433464+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:06:52.433464+0000","last_undegraded":"2024-09-21T02:06:52.433464+0000","last_fullsized":"2024-09-21T02:06:52.433464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427717+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:06:52.427717+0000","last_peered":"2024-09-21T02:06:52.427717+0000","last_clean":"2024-09-21T02:06:52.427717+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:06:52.427717+0000","last_undegraded":"2024-09-21T02:06:52.427717+0000","last_fullsized":"2024-09-21T02:06:52.427717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431461+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:06:52.431461+0000","last_peered":"2024-09-21T02:06:52.431461+0000","last_clean":"2024-09-21T02:06:52.431461+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:06:52.431461+0000","last_undegraded":"2024-09-21T02:06:52.431461+0000","last_fullsized":"2024-09-21T02:06:52.431461+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":471252,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":1159,"num_read_kb":672,"num_write":337,"num_write_kb":904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":675840,"data_stored":1028118,"data_compressed":37018,"data_compressed_allocated":49152,"data_compressed_original":761856,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710765,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":501553,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874289,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":1155211,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037815,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":504442,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":893828,"data_compressed":48968,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1099905,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561095,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":496875,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1093437,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":500949,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117078,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":495558,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378716,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":565992,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542241,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":569320,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":498331,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":74192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397621,"data_compressed":17729,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394604,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:07:13.485 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T02:07:13.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T02:07:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:13 smithi123 ceph-mon[33453]: pgmap v354: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T02:07:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:13 smithi123 ceph-mon[33453]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:13.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:13 smithi161 ceph-mon[34061]: pgmap v354: 129 pgs: 129 active+clean; 1.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 62 KiB/s wr, 17 op/s 2024-09-21T02:07:13.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:13 smithi161 ceph-mon[34061]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:13.748 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 -- 172.21.15.16:0/3101937018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 msgr2=0x7f66941087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 --2- 172.21.15.16:0/3101937018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941087b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6684009900 tx=0x7f668402a410 comp rx=0 tx=0).stop 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 -- 172.21.15.16:0/3101937018 shutdown_connections 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 --2- 172.21.15.16:0/3101937018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 0x7f669410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 --2- 172.21.15.16:0/3101937018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941087b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 --2- 172.21.15.16:0/3101937018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 0x7f6694102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.245+0000 7f669c0e9640 1 -- 172.21.15.16:0/3101937018 >> 172.21.15.16:0/3101937018 conn(0x7f6694079730 msgr2=0x7f6694079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- 172.21.15.16:0/3101937018 shutdown_connections 2024-09-21T02:07:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- 172.21.15.16:0/3101937018 wait complete. 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 Processor -- start 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- start start 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 0x7f669410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 0x7f6694110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6694113910 con 0x7f66941083a0 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6694113790 con 0x7f6694102a60 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.246+0000 7f669c0e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6694113a90 con 0x7f6694108cf0 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57722/0 (socket says 172.21.15.16:57722) 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 -- 172.21.15.16:0/1342936060 learned_addr learned my addr 172.21.15.16:0/1342936060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f669a686640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 0x7f6694110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699e85640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 0x7f669410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 -- 172.21.15.16:0/1342936060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 msgr2=0x7f6694110630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 0x7f6694110630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 -- 172.21.15.16:0/1342936060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 msgr2=0x7f669410fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 0x7f669410fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 -- 172.21.15.16:0/1342936060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66941a8660 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f6699684640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6684005940 tx=0x7f66840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668403f030 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66941a88f0 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6684003c90 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684035070 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.247+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66941a8e20 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.248+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6694102e70 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.250+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f6684042030 con 0x7f66941083a0 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.251+0000 7f668affd640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 0x7f66800b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.251+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f6684108360 con 0x7f66941083a0 2024-09-21T02:07:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.251+0000 7f6699e85640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 0x7f66800b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.251+0000 7f6699e85640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 0x7f66800b8b00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6690005fd0 tx=0x7f6690005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.254+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f66840ce600 con 0x7f66941083a0 2024-09-21T02:07:14.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.440+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6694063800 con 0x7f66800b65f0 2024-09-21T02:07:14.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.446+0000 7f668affd640 1 -- 172.21.15.16:0/1342936060 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340921 (secure 0 0 0) 0x7f6694063800 con 0x7f66800b65f0 2024-09-21T02:07:14.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:14.451 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 msgr2=0x7f66800b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 0x7f66800b8b00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6690005fd0 tx=0x7f6690005850 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 msgr2=0x7f66941100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6684005940 tx=0x7f66840042f0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 shutdown_connections 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66800b65f0 0x7f66800b8b00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6694108cf0 0x7f6694110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66941083a0 0x7f66941100f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 --2- 172.21.15.16:0/1342936060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6694102a60 0x7f669410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:14.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 >> 172.21.15.16:0/1342936060 conn(0x7f6694079730 msgr2=0x7f66941044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:14.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 shutdown_connections 2024-09-21T02:07:14.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:14.451+0000 7f669c0e9640 1 -- 172.21.15.16:0/1342936060 wait complete. 2024-09-21T02:07:15.191 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-21T02:07:14.060542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1459419,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1591,"num_read_kb":1047,"num_write":626,"num_write_kb":4412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":975,"ondisk_log_size":975,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":394472,"kb_used_data":11932,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124547352,"statfs":{"total":1151940427776,"available":1151536488448,"internally_reserved":0,"allocated":12218368,"data_stored":10347510,"data_compressed":577350,"data_compressed_allocated":774144,"data_compressed_original":5038080,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":793806,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":475,"num_read_kb":379,"num_write":237,"num_write_kb":2102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003764"},"pg_stats":[{"pgid":"2.1f","version":"86'14","reported_seq":167,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:10.907203+0000","last_change":"2024-09-21T02:04:31.619986+0000","last_active":"2024-09-21T02:07:10.907203+0000","last_peered":"2024-09-21T02:07:10.907203+0000","last_clean":"2024-09-21T02:07:10.907203+0000","last_became_active":"2024-09-21T02:04:31.619622+0000","last_became_peered":"2024-09-21T02:04:31.619622+0000","last_unstale":"2024-09-21T02:07:10.907203+0000","last_undegraded":"2024-09-21T02:07:10.907203+0000","last_fullsized":"2024-09-21T02:07:10.907203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:21.221189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435128+0000","last_change":"2024-09-21T02:04:33.637296+0000","last_active":"2024-09-21T02:06:52.435128+0000","last_peered":"2024-09-21T02:06:52.435128+0000","last_clean":"2024-09-21T02:06:52.435128+0000","last_became_active":"2024-09-21T02:04:33.636259+0000","last_became_peered":"2024-09-21T02:04:33.636259+0000","last_unstale":"2024-09-21T02:06:52.435128+0000","last_undegraded":"2024-09-21T02:06:52.435128+0000","last_fullsized":"2024-09-21T02:06:52.435128+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.678277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309551+0000","last_change":"2024-09-21T02:04:40.694862+0000","last_active":"2024-09-21T02:05:54.309551+0000","last_peered":"2024-09-21T02:05:54.309551+0000","last_clean":"2024-09-21T02:05:54.309551+0000","last_became_active":"2024-09-21T02:04:40.694419+0000","last_became_peered":"2024-09-21T02:04:40.694419+0000","last_unstale":"2024-09-21T02:05:54.309551+0000","last_undegraded":"2024-09-21T02:05:54.309551+0000","last_fullsized":"2024-09-21T02:05:54.309551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:36.262207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.694353+0000","last_change":"2024-09-21T02:04:31.824330+0000","last_active":"2024-09-21T02:05:54.694353+0000","last_peered":"2024-09-21T02:05:54.694353+0000","last_clean":"2024-09-21T02:05:54.694353+0000","last_became_active":"2024-09-21T02:04:31.823872+0000","last_became_peered":"2024-09-21T02:04:31.823872+0000","last_unstale":"2024-09-21T02:05:54.694353+0000","last_undegraded":"2024-09-21T02:05:54.694353+0000","last_fullsized":"2024-09-21T02:05:54.694353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:01:51.732339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434557+0000","last_change":"2024-09-21T02:04:33.629352+0000","last_active":"2024-09-21T02:06:52.434557+0000","last_peered":"2024-09-21T02:06:52.434557+0000","last_clean":"2024-09-21T02:06:52.434557+0000","last_became_active":"2024-09-21T02:04:33.628939+0000","last_became_peered":"2024-09-21T02:04:33.628939+0000","last_unstale":"2024-09-21T02:06:52.434557+0000","last_undegraded":"2024-09-21T02:06:52.434557+0000","last_fullsized":"2024-09-21T02:06:52.434557+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:25.208716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313872+0000","last_change":"2024-09-21T02:04:40.700854+0000","last_active":"2024-09-21T02:05:54.313872+0000","last_peered":"2024-09-21T02:05:54.313872+0000","last_clean":"2024-09-21T02:05:54.313872+0000","last_became_active":"2024-09-21T02:04:40.700079+0000","last_became_peered":"2024-09-21T02:04:40.700079+0000","last_unstale":"2024-09-21T02:05:54.313872+0000","last_undegraded":"2024-09-21T02:05:54.313872+0000","last_fullsized":"2024-09-21T02:05:54.313872+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:16:55.756525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314180+0000","last_change":"2024-09-21T02:04:53.754465+0000","last_active":"2024-09-21T02:05:54.314180+0000","last_peered":"2024-09-21T02:05:54.314180+0000","last_clean":"2024-09-21T02:05:54.314180+0000","last_became_active":"2024-09-21T02:04:53.746991+0000","last_became_peered":"2024-09-21T02:04:53.746991+0000","last_unstale":"2024-09-21T02:05:54.314180+0000","last_undegraded":"2024-09-21T02:05:54.314180+0000","last_fullsized":"2024-09-21T02:05:54.314180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:18.510883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447667+0000","last_change":"2024-09-21T02:04:33.632999+0000","last_active":"2024-09-21T02:06:52.447667+0000","last_peered":"2024-09-21T02:06:52.447667+0000","last_clean":"2024-09-21T02:06:52.447667+0000","last_became_active":"2024-09-21T02:04:33.632645+0000","last_became_peered":"2024-09-21T02:04:33.632645+0000","last_unstale":"2024-09-21T02:06:52.447667+0000","last_undegraded":"2024-09-21T02:06:52.447667+0000","last_fullsized":"2024-09-21T02:06:52.447667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:30:24.342776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.841631+0000","last_change":"2024-09-21T02:05:54.841083+0000","last_active":"2024-09-21T02:05:54.841631+0000","last_peered":"2024-09-21T02:05:54.841631+0000","last_clean":"2024-09-21T02:05:54.841631+0000","last_became_active":"2024-09-21T02:05:54.840645+0000","last_became_peered":"2024-09-21T02:05:54.840645+0000","last_unstale":"2024-09-21T02:05:54.841631+0000","last_undegraded":"2024-09-21T02:05:54.841631+0000","last_fullsized":"2024-09-21T02:05:54.841631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:33:35.107986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,8],"acting":[4,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.840304+0000","last_change":"2024-09-21T02:04:31.622208+0000","last_active":"2024-09-21T02:05:54.840304+0000","last_peered":"2024-09-21T02:05:54.840304+0000","last_clean":"2024-09-21T02:05:54.840304+0000","last_became_active":"2024-09-21T02:04:31.621163+0000","last_became_peered":"2024-09-21T02:04:31.621163+0000","last_unstale":"2024-09-21T02:05:54.840304+0000","last_undegraded":"2024-09-21T02:05:54.840304+0000","last_fullsized":"2024-09-21T02:05:54.840304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:20:06.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445936+0000","last_change":"2024-09-21T02:04:54.142542+0000","last_active":"2024-09-21T02:06:52.445936+0000","last_peered":"2024-09-21T02:06:52.445936+0000","last_clean":"2024-09-21T02:06:52.445936+0000","last_became_active":"2024-09-21T02:04:54.142134+0000","last_became_peered":"2024-09-21T02:04:54.142134+0000","last_unstale":"2024-09-21T02:06:52.445936+0000","last_undegraded":"2024-09-21T02:06:52.445936+0000","last_fullsized":"2024-09-21T02:06:52.445936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:08:52.385486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.313994+0000","last_change":"2024-09-21T02:04:40.694317+0000","last_active":"2024-09-21T02:05:54.313994+0000","last_peered":"2024-09-21T02:05:54.313994+0000","last_clean":"2024-09-21T02:05:54.313994+0000","last_became_active":"2024-09-21T02:04:40.693775+0000","last_became_peered":"2024-09-21T02:04:40.693775+0000","last_unstale":"2024-09-21T02:05:54.313994+0000","last_undegraded":"2024-09-21T02:05:54.313994+0000","last_fullsized":"2024-09-21T02:05:54.313994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:03.431855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445821+0000","last_change":"2024-09-21T02:04:33.631463+0000","last_active":"2024-09-21T02:06:52.445821+0000","last_peered":"2024-09-21T02:06:52.445821+0000","last_clean":"2024-09-21T02:06:52.445821+0000","last_became_active":"2024-09-21T02:04:33.631035+0000","last_became_peered":"2024-09-21T02:04:33.631035+0000","last_unstale":"2024-09-21T02:06:52.445821+0000","last_undegraded":"2024-09-21T02:06:52.445821+0000","last_fullsized":"2024-09-21T02:06:52.445821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:25:22.311891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":56,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839948+0000","last_change":"2024-09-21T02:04:53.751990+0000","last_active":"2024-09-21T02:05:54.839948+0000","last_peered":"2024-09-21T02:05:54.839948+0000","last_clean":"2024-09-21T02:05:54.839948+0000","last_became_active":"2024-09-21T02:04:53.748197+0000","last_became_peered":"2024-09-21T02:04:53.748197+0000","last_unstale":"2024-09-21T02:05:54.839948+0000","last_undegraded":"2024-09-21T02:05:54.839948+0000","last_fullsized":"2024-09-21T02:05:54.839948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:31:40.400030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314223+0000","last_change":"2024-09-21T02:04:41.070425+0000","last_active":"2024-09-21T02:05:54.314223+0000","last_peered":"2024-09-21T02:05:54.314223+0000","last_clean":"2024-09-21T02:05:54.314223+0000","last_became_active":"2024-09-21T02:04:41.069334+0000","last_became_peered":"2024-09-21T02:04:41.069334+0000","last_unstale":"2024-09-21T02:05:54.314223+0000","last_undegraded":"2024-09-21T02:05:54.314223+0000","last_fullsized":"2024-09-21T02:05:54.314223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:39:34.215465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432917+0000","last_change":"2024-09-21T02:04:33.628884+0000","last_active":"2024-09-21T02:06:52.432917+0000","last_peered":"2024-09-21T02:06:52.432917+0000","last_clean":"2024-09-21T02:06:52.432917+0000","last_became_active":"2024-09-21T02:04:33.628552+0000","last_became_peered":"2024-09-21T02:04:33.628552+0000","last_unstale":"2024-09-21T02:06:52.432917+0000","last_undegraded":"2024-09-21T02:06:52.432917+0000","last_fullsized":"2024-09-21T02:06:52.432917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:13.505885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:41.387929+0000","last_change":"2024-09-21T02:04:31.620318+0000","last_active":"2024-09-21T02:06:41.387929+0000","last_peered":"2024-09-21T02:06:41.387929+0000","last_clean":"2024-09-21T02:06:41.387929+0000","last_became_active":"2024-09-21T02:04:31.619010+0000","last_became_peered":"2024-09-21T02:04:31.619010+0000","last_unstale":"2024-09-21T02:06:41.387929+0000","last_undegraded":"2024-09-21T02:06:41.387929+0000","last_fullsized":"2024-09-21T02:06:41.387929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:04.887252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314252+0000","last_change":"2024-09-21T02:04:40.700686+0000","last_active":"2024-09-21T02:05:54.314252+0000","last_peered":"2024-09-21T02:05:54.314252+0000","last_clean":"2024-09-21T02:05:54.314252+0000","last_became_active":"2024-09-21T02:04:40.700190+0000","last_became_peered":"2024-09-21T02:04:40.700190+0000","last_unstale":"2024-09-21T02:05:54.314252+0000","last_undegraded":"2024-09-21T02:05:54.314252+0000","last_fullsized":"2024-09-21T02:05:54.314252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:50:41.116842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428361+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.428361+0000","last_peered":"2024-09-21T02:06:52.428361+0000","last_clean":"2024-09-21T02:06:52.428361+0000","last_became_active":"2024-09-21T02:04:33.631349+0000","last_became_peered":"2024-09-21T02:04:33.631349+0000","last_unstale":"2024-09-21T02:06:52.428361+0000","last_undegraded":"2024-09-21T02:06:52.428361+0000","last_fullsized":"2024-09-21T02:06:52.428361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:31.901065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693914+0000","last_change":"2024-09-21T02:04:31.619724+0000","last_active":"2024-09-21T02:05:54.693914+0000","last_peered":"2024-09-21T02:05:54.693914+0000","last_clean":"2024-09-21T02:05:54.693914+0000","last_became_active":"2024-09-21T02:04:31.619232+0000","last_became_peered":"2024-09-21T02:04:31.619232+0000","last_unstale":"2024-09-21T02:05:54.693914+0000","last_undegraded":"2024-09-21T02:05:54.693914+0000","last_fullsized":"2024-09-21T02:05:54.693914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:57:57.414099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.754062+0000","last_change":"2024-09-21T02:05:54.753929+0000","last_active":"2024-09-21T02:05:54.754062+0000","last_peered":"2024-09-21T02:05:54.754062+0000","last_clean":"2024-09-21T02:05:54.754062+0000","last_became_active":"2024-09-21T02:05:54.753431+0000","last_became_peered":"2024-09-21T02:05:54.753431+0000","last_unstale":"2024-09-21T02:05:54.754062+0000","last_undegraded":"2024-09-21T02:05:54.754062+0000","last_fullsized":"2024-09-21T02:05:54.754062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:20:41.311972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.391580+0000","last_change":"2024-09-21T02:04:33.627512+0000","last_active":"2024-09-21T02:06:52.391580+0000","last_peered":"2024-09-21T02:06:52.391580+0000","last_clean":"2024-09-21T02:06:52.391580+0000","last_became_active":"2024-09-21T02:04:33.627132+0000","last_became_peered":"2024-09-21T02:04:33.627132+0000","last_unstale":"2024-09-21T02:06:52.391580+0000","last_undegraded":"2024-09-21T02:06:52.391580+0000","last_fullsized":"2024-09-21T02:06:52.391580+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:23.209345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839098+0000","last_change":"2024-09-21T02:04:31.824802+0000","last_active":"2024-09-21T02:05:54.839098+0000","last_peered":"2024-09-21T02:05:54.839098+0000","last_clean":"2024-09-21T02:05:54.839098+0000","last_became_active":"2024-09-21T02:04:31.824352+0000","last_became_peered":"2024-09-21T02:04:31.824352+0000","last_unstale":"2024-09-21T02:05:54.839098+0000","last_undegraded":"2024-09-21T02:05:54.839098+0000","last_fullsized":"2024-09-21T02:05:54.839098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:01.876726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314296+0000","last_change":"2024-09-21T02:04:41.070452+0000","last_active":"2024-09-21T02:05:54.314296+0000","last_peered":"2024-09-21T02:05:54.314296+0000","last_clean":"2024-09-21T02:05:54.314296+0000","last_became_active":"2024-09-21T02:04:41.069779+0000","last_became_peered":"2024-09-21T02:04:41.069779+0000","last_unstale":"2024-09-21T02:05:54.314296+0000","last_undegraded":"2024-09-21T02:05:54.314296+0000","last_fullsized":"2024-09-21T02:05:54.314296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:18.007382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432105+0000","last_change":"2024-09-21T02:04:33.638864+0000","last_active":"2024-09-21T02:06:52.432105+0000","last_peered":"2024-09-21T02:06:52.432105+0000","last_clean":"2024-09-21T02:06:52.432105+0000","last_became_active":"2024-09-21T02:04:33.638074+0000","last_became_peered":"2024-09-21T02:04:33.638074+0000","last_unstale":"2024-09-21T02:06:52.432105+0000","last_undegraded":"2024-09-21T02:06:52.432105+0000","last_fullsized":"2024-09-21T02:06:52.432105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:26:18.438365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314615+0000","last_change":"2024-09-21T02:04:31.824031+0000","last_active":"2024-09-21T02:05:54.314615+0000","last_peered":"2024-09-21T02:05:54.314615+0000","last_clean":"2024-09-21T02:05:54.314615+0000","last_became_active":"2024-09-21T02:04:31.823663+0000","last_became_peered":"2024-09-21T02:04:31.823663+0000","last_unstale":"2024-09-21T02:05:54.314615+0000","last_undegraded":"2024-09-21T02:05:54.314615+0000","last_fullsized":"2024-09-21T02:05:54.314615+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:47.129316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309689+0000","last_change":"2024-09-21T02:04:40.688212+0000","last_active":"2024-09-21T02:05:54.309689+0000","last_peered":"2024-09-21T02:05:54.309689+0000","last_clean":"2024-09-21T02:05:54.309689+0000","last_became_active":"2024-09-21T02:04:40.687717+0000","last_became_peered":"2024-09-21T02:04:40.687717+0000","last_unstale":"2024-09-21T02:05:54.309689+0000","last_undegraded":"2024-09-21T02:05:54.309689+0000","last_fullsized":"2024-09-21T02:05:54.309689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:24.369688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437884+0000","last_change":"2024-09-21T02:04:33.634023+0000","last_active":"2024-09-21T02:06:52.437884+0000","last_peered":"2024-09-21T02:06:52.437884+0000","last_clean":"2024-09-21T02:06:52.437884+0000","last_became_active":"2024-09-21T02:04:33.633654+0000","last_became_peered":"2024-09-21T02:04:33.633654+0000","last_unstale":"2024-09-21T02:06:52.437884+0000","last_undegraded":"2024-09-21T02:06:52.437884+0000","last_fullsized":"2024-09-21T02:06:52.437884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:26:42.538426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'102","reported_seq":231,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:09.851842+0000","last_change":"2024-09-21T02:04:31.616709+0000","last_active":"2024-09-21T02:07:09.851842+0000","last_peered":"2024-09-21T02:07:09.851842+0000","last_clean":"2024-09-21T02:07:09.851842+0000","last_became_active":"2024-09-21T02:04:31.616309+0000","last_became_peered":"2024-09-21T02:04:31.616309+0000","last_unstale":"2024-09-21T02:07:09.851842+0000","last_undegraded":"2024-09-21T02:07:09.851842+0000","last_fullsized":"2024-09-21T02:07:09.851842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:02:05.646138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":186,"num_write_kb":1222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693575+0000","last_change":"2024-09-21T02:04:41.069276+0000","last_active":"2024-09-21T02:05:54.693575+0000","last_peered":"2024-09-21T02:05:54.693575+0000","last_clean":"2024-09-21T02:05:54.693575+0000","last_became_active":"2024-09-21T02:04:41.068798+0000","last_became_peered":"2024-09-21T02:04:41.068798+0000","last_unstale":"2024-09-21T02:05:54.693575+0000","last_undegraded":"2024-09-21T02:05:54.693575+0000","last_fullsized":"2024-09-21T02:05:54.693575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:47:02.847847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430290+0000","last_change":"2024-09-21T02:04:33.619970+0000","last_active":"2024-09-21T02:06:52.430290+0000","last_peered":"2024-09-21T02:06:52.430290+0000","last_clean":"2024-09-21T02:06:52.430290+0000","last_became_active":"2024-09-21T02:04:33.619169+0000","last_became_peered":"2024-09-21T02:04:33.619169+0000","last_unstale":"2024-09-21T02:06:52.430290+0000","last_undegraded":"2024-09-21T02:06:52.430290+0000","last_fullsized":"2024-09-21T02:06:52.430290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:56:42.097513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314156+0000","last_change":"2024-09-21T02:04:53.747753+0000","last_active":"2024-09-21T02:05:54.314156+0000","last_peered":"2024-09-21T02:05:54.314156+0000","last_clean":"2024-09-21T02:05:54.314156+0000","last_became_active":"2024-09-21T02:04:53.746849+0000","last_became_peered":"2024-09-21T02:04:53.746849+0000","last_unstale":"2024-09-21T02:05:54.314156+0000","last_undegraded":"2024-09-21T02:05:54.314156+0000","last_fullsized":"2024-09-21T02:05:54.314156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:43:34.571517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314000+0000","last_change":"2024-09-21T02:04:40.694517+0000","last_active":"2024-09-21T02:05:54.314000+0000","last_peered":"2024-09-21T02:05:54.314000+0000","last_clean":"2024-09-21T02:05:54.314000+0000","last_became_active":"2024-09-21T02:04:40.693998+0000","last_became_peered":"2024-09-21T02:04:40.693998+0000","last_unstale":"2024-09-21T02:05:54.314000+0000","last_undegraded":"2024-09-21T02:05:54.314000+0000","last_fullsized":"2024-09-21T02:05:54.314000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:02.947713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431581+0000","last_change":"2024-09-21T02:04:33.631732+0000","last_active":"2024-09-21T02:06:52.431581+0000","last_peered":"2024-09-21T02:06:52.431581+0000","last_clean":"2024-09-21T02:06:52.431581+0000","last_became_active":"2024-09-21T02:04:33.630858+0000","last_became_peered":"2024-09-21T02:04:33.630858+0000","last_unstale":"2024-09-21T02:06:52.431581+0000","last_undegraded":"2024-09-21T02:06:52.431581+0000","last_fullsized":"2024-09-21T02:06:52.431581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:32.820065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796090+0000","last_change":"2024-09-21T02:04:31.824766+0000","last_active":"2024-09-21T02:05:54.796090+0000","last_peered":"2024-09-21T02:05:54.796090+0000","last_clean":"2024-09-21T02:05:54.796090+0000","last_became_active":"2024-09-21T02:04:31.824333+0000","last_became_peered":"2024-09-21T02:04:31.824333+0000","last_unstale":"2024-09-21T02:05:54.796090+0000","last_undegraded":"2024-09-21T02:05:54.796090+0000","last_fullsized":"2024-09-21T02:05:54.796090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:26:07.854857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839552+0000","last_change":"2024-09-21T02:04:41.069833+0000","last_active":"2024-09-21T02:05:54.839552+0000","last_peered":"2024-09-21T02:05:54.839552+0000","last_clean":"2024-09-21T02:05:54.839552+0000","last_became_active":"2024-09-21T02:04:41.069317+0000","last_became_peered":"2024-09-21T02:04:41.069317+0000","last_unstale":"2024-09-21T02:05:54.839552+0000","last_undegraded":"2024-09-21T02:05:54.839552+0000","last_fullsized":"2024-09-21T02:05:54.839552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:09:55.160645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427901+0000","last_change":"2024-09-21T02:04:33.634154+0000","last_active":"2024-09-21T02:06:52.427901+0000","last_peered":"2024-09-21T02:06:52.427901+0000","last_clean":"2024-09-21T02:06:52.427901+0000","last_became_active":"2024-09-21T02:04:33.629896+0000","last_became_peered":"2024-09-21T02:04:33.629896+0000","last_unstale":"2024-09-21T02:06:52.427901+0000","last_undegraded":"2024-09-21T02:06:52.427901+0000","last_fullsized":"2024-09-21T02:06:52.427901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:33:43.541741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'8","reported_seq":162,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.905124+0000","last_change":"2024-09-21T02:04:31.621075+0000","last_active":"2024-09-21T02:07:08.905124+0000","last_peered":"2024-09-21T02:07:08.905124+0000","last_clean":"2024-09-21T02:07:08.905124+0000","last_became_active":"2024-09-21T02:04:31.620677+0000","last_became_peered":"2024-09-21T02:04:31.620677+0000","last_unstale":"2024-09-21T02:07:08.905124+0000","last_undegraded":"2024-09-21T02:07:08.905124+0000","last_fullsized":"2024-09-21T02:07:08.905124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:52:55.821531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308834+0000","last_change":"2024-09-21T02:04:40.689771+0000","last_active":"2024-09-21T02:05:54.308834+0000","last_peered":"2024-09-21T02:05:54.308834+0000","last_clean":"2024-09-21T02:05:54.308834+0000","last_became_active":"2024-09-21T02:04:40.689324+0000","last_became_peered":"2024-09-21T02:04:40.689324+0000","last_unstale":"2024-09-21T02:05:54.308834+0000","last_undegraded":"2024-09-21T02:05:54.308834+0000","last_fullsized":"2024-09-21T02:05:54.308834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:22:54.054746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431202+0000","last_change":"2024-09-21T02:04:33.634452+0000","last_active":"2024-09-21T02:06:52.431202+0000","last_peered":"2024-09-21T02:06:52.431202+0000","last_clean":"2024-09-21T02:06:52.431202+0000","last_became_active":"2024-09-21T02:04:33.633676+0000","last_became_peered":"2024-09-21T02:04:33.633676+0000","last_unstale":"2024-09-21T02:06:52.431202+0000","last_undegraded":"2024-09-21T02:06:52.431202+0000","last_fullsized":"2024-09-21T02:06:52.431202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:25:27.860783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":157,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:08.904571+0000","last_change":"2024-09-21T02:04:31.824221+0000","last_active":"2024-09-21T02:07:08.904571+0000","last_peered":"2024-09-21T02:07:08.904571+0000","last_clean":"2024-09-21T02:07:08.904571+0000","last_became_active":"2024-09-21T02:04:31.823798+0000","last_became_peered":"2024-09-21T02:04:31.823798+0000","last_unstale":"2024-09-21T02:07:08.904571+0000","last_undegraded":"2024-09-21T02:07:08.904571+0000","last_fullsized":"2024-09-21T02:07:08.904571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:42.394234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309231+0000","last_change":"2024-09-21T02:04:41.069326+0000","last_active":"2024-09-21T02:05:54.309231+0000","last_peered":"2024-09-21T02:05:54.309231+0000","last_clean":"2024-09-21T02:05:54.309231+0000","last_became_active":"2024-09-21T02:04:41.068890+0000","last_became_peered":"2024-09-21T02:04:41.068890+0000","last_unstale":"2024-09-21T02:05:54.309231+0000","last_undegraded":"2024-09-21T02:05:54.309231+0000","last_fullsized":"2024-09-21T02:05:54.309231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:42.680335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431015+0000","last_change":"2024-09-21T02:04:33.624613+0000","last_active":"2024-09-21T02:06:52.431015+0000","last_peered":"2024-09-21T02:06:52.431015+0000","last_clean":"2024-09-21T02:06:52.431015+0000","last_became_active":"2024-09-21T02:04:33.624266+0000","last_became_peered":"2024-09-21T02:04:33.624266+0000","last_unstale":"2024-09-21T02:06:52.431015+0000","last_undegraded":"2024-09-21T02:06:52.431015+0000","last_fullsized":"2024-09-21T02:06:52.431015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:56:38.278968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799870+0000","last_change":"2024-09-21T02:04:31.620428+0000","last_active":"2024-09-21T02:05:54.799870+0000","last_peered":"2024-09-21T02:05:54.799870+0000","last_clean":"2024-09-21T02:05:54.799870+0000","last_became_active":"2024-09-21T02:04:31.619503+0000","last_became_peered":"2024-09-21T02:04:31.619503+0000","last_unstale":"2024-09-21T02:05:54.799870+0000","last_undegraded":"2024-09-21T02:05:54.799870+0000","last_fullsized":"2024-09-21T02:05:54.799870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:50.551723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752633+0000","last_change":"2024-09-21T02:04:41.069918+0000","last_active":"2024-09-21T02:05:54.752633+0000","last_peered":"2024-09-21T02:05:54.752633+0000","last_clean":"2024-09-21T02:05:54.752633+0000","last_became_active":"2024-09-21T02:04:41.069362+0000","last_became_peered":"2024-09-21T02:04:41.069362+0000","last_unstale":"2024-09-21T02:05:54.752633+0000","last_undegraded":"2024-09-21T02:05:54.752633+0000","last_fullsized":"2024-09-21T02:05:54.752633+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:21.159295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436058+0000","last_change":"2024-09-21T02:04:33.635604+0000","last_active":"2024-09-21T02:06:52.436058+0000","last_peered":"2024-09-21T02:06:52.436058+0000","last_clean":"2024-09-21T02:06:52.436058+0000","last_became_active":"2024-09-21T02:04:33.635267+0000","last_became_peered":"2024-09-21T02:04:33.635267+0000","last_unstale":"2024-09-21T02:06:52.436058+0000","last_undegraded":"2024-09-21T02:06:52.436058+0000","last_fullsized":"2024-09-21T02:06:52.436058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:42.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314486+0000","last_change":"2024-09-21T02:04:31.625686+0000","last_active":"2024-09-21T02:05:54.314486+0000","last_peered":"2024-09-21T02:05:54.314486+0000","last_clean":"2024-09-21T02:05:54.314486+0000","last_became_active":"2024-09-21T02:04:31.625187+0000","last_became_peered":"2024-09-21T02:04:31.625187+0000","last_unstale":"2024-09-21T02:05:54.314486+0000","last_undegraded":"2024-09-21T02:05:54.314486+0000","last_fullsized":"2024-09-21T02:05:54.314486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:39:11.310431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799508+0000","last_change":"2024-09-21T02:03:20.526663+0000","last_active":"2024-09-21T02:05:54.799508+0000","last_peered":"2024-09-21T02:05:54.799508+0000","last_clean":"2024-09-21T02:05:54.799508+0000","last_became_active":"2024-09-21T02:03:20.519903+0000","last_became_peered":"2024-09-21T02:03:20.519903+0000","last_unstale":"2024-09-21T02:05:54.799508+0000","last_undegraded":"2024-09-21T02:05:54.799508+0000","last_fullsized":"2024-09-21T02:05:54.799508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:59:53.286035+0000","last_clean_scrub_stamp":"2024-09-21T01:59:53.286035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:13:09.463996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799546+0000","last_change":"2024-09-21T02:04:40.693338+0000","last_active":"2024-09-21T02:05:54.799546+0000","last_peered":"2024-09-21T02:05:54.799546+0000","last_clean":"2024-09-21T02:05:54.799546+0000","last_became_active":"2024-09-21T02:04:40.692819+0000","last_became_peered":"2024-09-21T02:04:40.692819+0000","last_unstale":"2024-09-21T02:05:54.799546+0000","last_undegraded":"2024-09-21T02:05:54.799546+0000","last_fullsized":"2024-09-21T02:05:54.799546+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:57:19.850906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444281+0000","last_change":"2024-09-21T02:04:33.635954+0000","last_active":"2024-09-21T02:06:52.444281+0000","last_peered":"2024-09-21T02:06:52.444281+0000","last_clean":"2024-09-21T02:06:52.444281+0000","last_became_active":"2024-09-21T02:04:33.635103+0000","last_became_peered":"2024-09-21T02:04:33.635103+0000","last_unstale":"2024-09-21T02:06:52.444281+0000","last_undegraded":"2024-09-21T02:06:52.444281+0000","last_fullsized":"2024-09-21T02:06:52.444281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:54.839582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'8","reported_seq":164,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:09.920672+0000","last_change":"2024-09-21T02:04:31.824651+0000","last_active":"2024-09-21T02:07:09.920672+0000","last_peered":"2024-09-21T02:07:09.920672+0000","last_clean":"2024-09-21T02:07:09.920672+0000","last_became_active":"2024-09-21T02:04:31.824224+0000","last_became_peered":"2024-09-21T02:04:31.824224+0000","last_unstale":"2024-09-21T02:07:09.920672+0000","last_undegraded":"2024-09-21T02:07:09.920672+0000","last_fullsized":"2024-09-21T02:07:09.920672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:19:02.739734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2195,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838953+0000","last_change":"2024-09-21T02:04:40.690500+0000","last_active":"2024-09-21T02:05:54.838953+0000","last_peered":"2024-09-21T02:05:54.838953+0000","last_clean":"2024-09-21T02:05:54.838953+0000","last_became_active":"2024-09-21T02:04:40.690015+0000","last_became_peered":"2024-09-21T02:04:40.690015+0000","last_unstale":"2024-09-21T02:05:54.838953+0000","last_undegraded":"2024-09-21T02:05:54.838953+0000","last_fullsized":"2024-09-21T02:05:54.838953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:15:25.200542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435516+0000","last_change":"2024-09-21T02:04:53.749505+0000","last_active":"2024-09-21T02:06:52.435516+0000","last_peered":"2024-09-21T02:06:52.435516+0000","last_clean":"2024-09-21T02:06:52.435516+0000","last_became_active":"2024-09-21T02:04:53.749029+0000","last_became_peered":"2024-09-21T02:04:53.749029+0000","last_unstale":"2024-09-21T02:06:52.435516+0000","last_undegraded":"2024-09-21T02:06:52.435516+0000","last_fullsized":"2024-09-21T02:06:52.435516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:45:03.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'62","reported_seq":132,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:09.495235+0000","last_change":"2024-09-21T02:04:31.619866+0000","last_active":"2024-09-21T02:07:09.495235+0000","last_peered":"2024-09-21T02:07:09.495235+0000","last_clean":"2024-09-21T02:07:09.495235+0000","last_became_active":"2024-09-21T02:04:31.619420+0000","last_became_peered":"2024-09-21T02:04:31.619420+0000","last_unstale":"2024-09-21T02:07:09.495235+0000","last_undegraded":"2024-09-21T02:07:09.495235+0000","last_fullsized":"2024-09-21T02:07:09.495235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:27.733307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838590+0000","last_change":"2024-09-21T02:04:40.696594+0000","last_active":"2024-09-21T02:05:54.838590+0000","last_peered":"2024-09-21T02:05:54.838590+0000","last_clean":"2024-09-21T02:05:54.838590+0000","last_became_active":"2024-09-21T02:04:40.696134+0000","last_became_peered":"2024-09-21T02:04:40.696134+0000","last_unstale":"2024-09-21T02:05:54.838590+0000","last_undegraded":"2024-09-21T02:05:54.838590+0000","last_fullsized":"2024-09-21T02:05:54.838590+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:41:29.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.443855+0000","last_change":"2024-09-21T02:04:33.625790+0000","last_active":"2024-09-21T02:06:52.443855+0000","last_peered":"2024-09-21T02:06:52.443855+0000","last_clean":"2024-09-21T02:06:52.443855+0000","last_became_active":"2024-09-21T02:04:33.625396+0000","last_became_peered":"2024-09-21T02:04:33.625396+0000","last_unstale":"2024-09-21T02:06:52.443855+0000","last_undegraded":"2024-09-21T02:06:52.443855+0000","last_fullsized":"2024-09-21T02:06:52.443855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:47.430270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309244+0000","last_change":"2024-09-21T02:04:31.614473+0000","last_active":"2024-09-21T02:05:54.309244+0000","last_peered":"2024-09-21T02:05:54.309244+0000","last_clean":"2024-09-21T02:05:54.309244+0000","last_became_active":"2024-09-21T02:04:31.613355+0000","last_became_peered":"2024-09-21T02:04:31.613355+0000","last_unstale":"2024-09-21T02:05:54.309244+0000","last_undegraded":"2024-09-21T02:05:54.309244+0000","last_fullsized":"2024-09-21T02:05:54.309244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:44.998504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751778+0000","last_change":"2024-09-21T02:04:40.690207+0000","last_active":"2024-09-21T02:05:54.751778+0000","last_peered":"2024-09-21T02:05:54.751778+0000","last_clean":"2024-09-21T02:05:54.751778+0000","last_became_active":"2024-09-21T02:04:40.689736+0000","last_became_peered":"2024-09-21T02:04:40.689736+0000","last_unstale":"2024-09-21T02:05:54.751778+0000","last_undegraded":"2024-09-21T02:05:54.751778+0000","last_fullsized":"2024-09-21T02:05:54.751778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:14:14.170943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436869+0000","last_change":"2024-09-21T02:04:33.637467+0000","last_active":"2024-09-21T02:06:52.436869+0000","last_peered":"2024-09-21T02:06:52.436869+0000","last_clean":"2024-09-21T02:06:52.436869+0000","last_became_active":"2024-09-21T02:04:33.636279+0000","last_became_peered":"2024-09-21T02:04:33.636279+0000","last_unstale":"2024-09-21T02:06:52.436869+0000","last_undegraded":"2024-09-21T02:06:52.436869+0000","last_fullsized":"2024-09-21T02:06:52.436869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:04.374277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'3","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:45.953185+0000","last_change":"2024-09-21T02:04:31.616525+0000","last_active":"2024-09-21T02:06:45.953185+0000","last_peered":"2024-09-21T02:06:45.953185+0000","last_clean":"2024-09-21T02:06:45.953185+0000","last_became_active":"2024-09-21T02:04:31.615958+0000","last_became_peered":"2024-09-21T02:04:31.615958+0000","last_unstale":"2024-09-21T02:06:45.953185+0000","last_undegraded":"2024-09-21T02:06:45.953185+0000","last_fullsized":"2024-09-21T02:06:45.953185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:35:43.091926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.751987+0000","last_change":"2024-09-21T02:04:40.697416+0000","last_active":"2024-09-21T02:05:54.751987+0000","last_peered":"2024-09-21T02:05:54.751987+0000","last_clean":"2024-09-21T02:05:54.751987+0000","last_became_active":"2024-09-21T02:04:40.694468+0000","last_became_peered":"2024-09-21T02:04:40.694468+0000","last_unstale":"2024-09-21T02:05:54.751987+0000","last_undegraded":"2024-09-21T02:05:54.751987+0000","last_fullsized":"2024-09-21T02:05:54.751987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:59:33.736771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437619+0000","last_change":"2024-09-21T02:04:54.143664+0000","last_active":"2024-09-21T02:06:52.437619+0000","last_peered":"2024-09-21T02:06:52.437619+0000","last_clean":"2024-09-21T02:06:52.437619+0000","last_became_active":"2024-09-21T02:04:54.143261+0000","last_became_peered":"2024-09-21T02:04:54.143261+0000","last_unstale":"2024-09-21T02:06:52.437619+0000","last_undegraded":"2024-09-21T02:06:52.437619+0000","last_fullsized":"2024-09-21T02:06:52.437619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:17:16.567025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839608+0000","last_change":"2024-09-21T02:04:54.150560+0000","last_active":"2024-09-21T02:05:54.839608+0000","last_peered":"2024-09-21T02:05:54.839608+0000","last_clean":"2024-09-21T02:05:54.839608+0000","last_became_active":"2024-09-21T02:04:54.142744+0000","last_became_peered":"2024-09-21T02:04:54.142744+0000","last_unstale":"2024-09-21T02:05:54.839608+0000","last_undegraded":"2024-09-21T02:05:54.839608+0000","last_fullsized":"2024-09-21T02:05:54.839608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:14.169830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309217+0000","last_change":"2024-09-21T02:04:40.694117+0000","last_active":"2024-09-21T02:05:54.309217+0000","last_peered":"2024-09-21T02:05:54.309217+0000","last_clean":"2024-09-21T02:05:54.309217+0000","last_became_active":"2024-09-21T02:04:40.693649+0000","last_became_peered":"2024-09-21T02:04:40.693649+0000","last_unstale":"2024-09-21T02:05:54.309217+0000","last_undegraded":"2024-09-21T02:05:54.309217+0000","last_fullsized":"2024-09-21T02:05:54.309217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:15.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445290+0000","last_change":"2024-09-21T02:04:33.633344+0000","last_active":"2024-09-21T02:06:52.445290+0000","last_peered":"2024-09-21T02:06:52.445290+0000","last_clean":"2024-09-21T02:06:52.445290+0000","last_became_active":"2024-09-21T02:04:33.632985+0000","last_became_peered":"2024-09-21T02:04:33.632985+0000","last_unstale":"2024-09-21T02:06:52.445290+0000","last_undegraded":"2024-09-21T02:06:52.445290+0000","last_fullsized":"2024-09-21T02:06:52.445290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:21.697683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'8","reported_seq":166,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:11.909505+0000","last_change":"2024-09-21T02:04:31.626929+0000","last_active":"2024-09-21T02:07:11.909505+0000","last_peered":"2024-09-21T02:07:11.909505+0000","last_clean":"2024-09-21T02:07:11.909505+0000","last_became_active":"2024-09-21T02:04:31.626569+0000","last_became_peered":"2024-09-21T02:04:31.626569+0000","last_unstale":"2024-09-21T02:07:11.909505+0000","last_undegraded":"2024-09-21T02:07:11.909505+0000","last_fullsized":"2024-09-21T02:07:11.909505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:40.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.796399+0000","last_change":"2024-09-21T02:04:40.698291+0000","last_active":"2024-09-21T02:05:54.796399+0000","last_peered":"2024-09-21T02:05:54.796399+0000","last_clean":"2024-09-21T02:05:54.796399+0000","last_became_active":"2024-09-21T02:04:40.697360+0000","last_became_peered":"2024-09-21T02:04:40.697360+0000","last_unstale":"2024-09-21T02:05:54.796399+0000","last_undegraded":"2024-09-21T02:05:54.796399+0000","last_fullsized":"2024-09-21T02:05:54.796399+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:29:35.346011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446922+0000","last_change":"2024-09-21T02:04:33.624823+0000","last_active":"2024-09-21T02:06:52.446922+0000","last_peered":"2024-09-21T02:06:52.446922+0000","last_clean":"2024-09-21T02:06:52.446922+0000","last_became_active":"2024-09-21T02:04:33.624474+0000","last_became_peered":"2024-09-21T02:04:33.624474+0000","last_unstale":"2024-09-21T02:06:52.446922+0000","last_undegraded":"2024-09-21T02:06:52.446922+0000","last_fullsized":"2024-09-21T02:06:52.446922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:08:43.609844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752428+0000","last_change":"2024-09-21T02:04:31.619999+0000","last_active":"2024-09-21T02:05:54.752428+0000","last_peered":"2024-09-21T02:05:54.752428+0000","last_clean":"2024-09-21T02:05:54.752428+0000","last_became_active":"2024-09-21T02:04:31.619592+0000","last_became_peered":"2024-09-21T02:04:31.619592+0000","last_unstale":"2024-09-21T02:05:54.752428+0000","last_undegraded":"2024-09-21T02:05:54.752428+0000","last_fullsized":"2024-09-21T02:05:54.752428+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:11.217249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314337+0000","last_change":"2024-09-21T02:04:40.695220+0000","last_active":"2024-09-21T02:05:54.314337+0000","last_peered":"2024-09-21T02:05:54.314337+0000","last_clean":"2024-09-21T02:05:54.314337+0000","last_became_active":"2024-09-21T02:04:40.694239+0000","last_became_peered":"2024-09-21T02:04:40.694239+0000","last_unstale":"2024-09-21T02:05:54.314337+0000","last_undegraded":"2024-09-21T02:05:54.314337+0000","last_fullsized":"2024-09-21T02:05:54.314337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:57.054140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446213+0000","last_change":"2024-09-21T02:04:33.632045+0000","last_active":"2024-09-21T02:06:52.446213+0000","last_peered":"2024-09-21T02:06:52.446213+0000","last_clean":"2024-09-21T02:06:52.446213+0000","last_became_active":"2024-09-21T02:04:33.631684+0000","last_became_peered":"2024-09-21T02:04:33.631684+0000","last_unstale":"2024-09-21T02:06:52.446213+0000","last_undegraded":"2024-09-21T02:06:52.446213+0000","last_fullsized":"2024-09-21T02:06:52.446213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:25:35.137083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309963+0000","last_change":"2024-09-21T02:04:31.620459+0000","last_active":"2024-09-21T02:05:54.309963+0000","last_peered":"2024-09-21T02:05:54.309963+0000","last_clean":"2024-09-21T02:05:54.309963+0000","last_became_active":"2024-09-21T02:04:31.620043+0000","last_became_peered":"2024-09-21T02:04:31.620043+0000","last_unstale":"2024-09-21T02:05:54.309963+0000","last_undegraded":"2024-09-21T02:05:54.309963+0000","last_fullsized":"2024-09-21T02:05:54.309963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:06:42.841305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752780+0000","last_change":"2024-09-21T02:04:40.690554+0000","last_active":"2024-09-21T02:05:54.752780+0000","last_peered":"2024-09-21T02:05:54.752780+0000","last_clean":"2024-09-21T02:05:54.752780+0000","last_became_active":"2024-09-21T02:04:40.689993+0000","last_became_peered":"2024-09-21T02:04:40.689993+0000","last_unstale":"2024-09-21T02:05:54.752780+0000","last_undegraded":"2024-09-21T02:05:54.752780+0000","last_fullsized":"2024-09-21T02:05:54.752780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:44:44.728022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.424275+0000","last_change":"2024-09-21T02:04:33.638289+0000","last_active":"2024-09-21T02:06:52.424275+0000","last_peered":"2024-09-21T02:06:52.424275+0000","last_clean":"2024-09-21T02:06:52.424275+0000","last_became_active":"2024-09-21T02:04:33.637898+0000","last_became_peered":"2024-09-21T02:04:33.637898+0000","last_unstale":"2024-09-21T02:06:52.424275+0000","last_undegraded":"2024-09-21T02:06:52.424275+0000","last_fullsized":"2024-09-21T02:06:52.424275+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:25:14.175366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'40","reported_seq":318,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:12.928977+0000","last_change":"2024-09-21T02:04:31.623745+0000","last_active":"2024-09-21T02:07:12.928977+0000","last_peered":"2024-09-21T02:07:12.928977+0000","last_clean":"2024-09-21T02:07:12.928977+0000","last_became_active":"2024-09-21T02:04:31.623372+0000","last_became_peered":"2024-09-21T02:04:31.623372+0000","last_unstale":"2024-09-21T02:07:12.928977+0000","last_undegraded":"2024-09-21T02:07:12.928977+0000","last_fullsized":"2024-09-21T02:07:12.928977+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:20.189402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":238,"num_read_kb":189,"num_write":42,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.795725+0000","last_change":"2024-09-21T02:04:40.696285+0000","last_active":"2024-09-21T02:05:54.795725+0000","last_peered":"2024-09-21T02:05:54.795725+0000","last_clean":"2024-09-21T02:05:54.795725+0000","last_became_active":"2024-09-21T02:04:40.695868+0000","last_became_peered":"2024-09-21T02:04:40.695868+0000","last_unstale":"2024-09-21T02:05:54.795725+0000","last_undegraded":"2024-09-21T02:05:54.795725+0000","last_fullsized":"2024-09-21T02:05:54.795725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:39:17.123413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444121+0000","last_change":"2024-09-21T02:04:33.638235+0000","last_active":"2024-09-21T02:06:52.444121+0000","last_peered":"2024-09-21T02:06:52.444121+0000","last_clean":"2024-09-21T02:06:52.444121+0000","last_became_active":"2024-09-21T02:04:33.637811+0000","last_became_peered":"2024-09-21T02:04:33.637811+0000","last_unstale":"2024-09-21T02:06:52.444121+0000","last_undegraded":"2024-09-21T02:06:52.444121+0000","last_fullsized":"2024-09-21T02:06:52.444121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:51.822165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752935+0000","last_change":"2024-09-21T02:04:31.825010+0000","last_active":"2024-09-21T02:05:54.752935+0000","last_peered":"2024-09-21T02:05:54.752935+0000","last_clean":"2024-09-21T02:05:54.752935+0000","last_became_active":"2024-09-21T02:04:31.824561+0000","last_became_peered":"2024-09-21T02:04:31.824561+0000","last_unstale":"2024-09-21T02:05:54.752935+0000","last_undegraded":"2024-09-21T02:05:54.752935+0000","last_fullsized":"2024-09-21T02:05:54.752935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:22:27.136690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.314193+0000","last_change":"2024-09-21T02:04:40.695387+0000","last_active":"2024-09-21T02:05:54.314193+0000","last_peered":"2024-09-21T02:05:54.314193+0000","last_clean":"2024-09-21T02:05:54.314193+0000","last_became_active":"2024-09-21T02:04:40.694845+0000","last_became_peered":"2024-09-21T02:04:40.694845+0000","last_unstale":"2024-09-21T02:05:54.314193+0000","last_undegraded":"2024-09-21T02:05:54.314193+0000","last_fullsized":"2024-09-21T02:05:54.314193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:26:08.776016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437003+0000","last_change":"2024-09-21T02:04:33.637205+0000","last_active":"2024-09-21T02:06:52.437003+0000","last_peered":"2024-09-21T02:06:52.437003+0000","last_clean":"2024-09-21T02:06:52.437003+0000","last_became_active":"2024-09-21T02:04:33.636860+0000","last_became_peered":"2024-09-21T02:04:33.636860+0000","last_unstale":"2024-09-21T02:06:52.437003+0000","last_undegraded":"2024-09-21T02:06:52.437003+0000","last_fullsized":"2024-09-21T02:06:52.437003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:29:30.211995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.799684+0000","last_change":"2024-09-21T02:04:31.622811+0000","last_active":"2024-09-21T02:05:54.799684+0000","last_peered":"2024-09-21T02:05:54.799684+0000","last_clean":"2024-09-21T02:05:54.799684+0000","last_became_active":"2024-09-21T02:04:31.622441+0000","last_became_peered":"2024-09-21T02:04:31.622441+0000","last_unstale":"2024-09-21T02:05:54.799684+0000","last_undegraded":"2024-09-21T02:05:54.799684+0000","last_fullsized":"2024-09-21T02:05:54.799684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:24.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.839390+0000","last_change":"2024-09-21T02:04:53.747894+0000","last_active":"2024-09-21T02:05:54.839390+0000","last_peered":"2024-09-21T02:05:54.839390+0000","last_clean":"2024-09-21T02:05:54.839390+0000","last_became_active":"2024-09-21T02:04:53.747383+0000","last_became_peered":"2024-09-21T02:04:53.747383+0000","last_unstale":"2024-09-21T02:05:54.839390+0000","last_undegraded":"2024-09-21T02:05:54.839390+0000","last_fullsized":"2024-09-21T02:05:54.839390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:23.169542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447850+0000","last_change":"2024-09-21T02:04:33.622578+0000","last_active":"2024-09-21T02:06:52.447850+0000","last_peered":"2024-09-21T02:06:52.447850+0000","last_clean":"2024-09-21T02:06:52.447850+0000","last_became_active":"2024-09-21T02:04:33.622215+0000","last_became_peered":"2024-09-21T02:04:33.622215+0000","last_unstale":"2024-09-21T02:06:52.447850+0000","last_undegraded":"2024-09-21T02:06:52.447850+0000","last_fullsized":"2024-09-21T02:06:52.447850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:19:04.977967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'6","reported_seq":272,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:12.911134+0000","last_change":"2024-09-21T02:04:31.621942+0000","last_active":"2024-09-21T02:07:12.911134+0000","last_peered":"2024-09-21T02:07:12.911134+0000","last_clean":"2024-09-21T02:07:12.911134+0000","last_became_active":"2024-09-21T02:04:31.621591+0000","last_became_peered":"2024-09-21T02:04:31.621591+0000","last_unstale":"2024-09-21T02:07:12.911134+0000","last_undegraded":"2024-09-21T02:07:12.911134+0000","last_fullsized":"2024-09-21T02:07:12.911134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:41:01.678687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.693641+0000","last_change":"2024-09-21T02:04:40.698822+0000","last_active":"2024-09-21T02:05:54.693641+0000","last_peered":"2024-09-21T02:05:54.693641+0000","last_clean":"2024-09-21T02:05:54.693641+0000","last_became_active":"2024-09-21T02:04:40.698417+0000","last_became_peered":"2024-09-21T02:04:40.698417+0000","last_unstale":"2024-09-21T02:05:54.693641+0000","last_undegraded":"2024-09-21T02:05:54.693641+0000","last_fullsized":"2024-09-21T02:05:54.693641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:46.462432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.838683+0000","last_change":"2024-09-21T02:04:40.688657+0000","last_active":"2024-09-21T02:05:54.838683+0000","last_peered":"2024-09-21T02:05:54.838683+0000","last_clean":"2024-09-21T02:05:54.838683+0000","last_became_active":"2024-09-21T02:04:40.688210+0000","last_became_peered":"2024-09-21T02:04:40.688210+0000","last_unstale":"2024-09-21T02:05:54.838683+0000","last_undegraded":"2024-09-21T02:05:54.838683+0000","last_fullsized":"2024-09-21T02:05:54.838683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:37.197333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429092+0000","last_change":"2024-09-21T02:04:33.627278+0000","last_active":"2024-09-21T02:06:52.429092+0000","last_peered":"2024-09-21T02:06:52.429092+0000","last_clean":"2024-09-21T02:06:52.429092+0000","last_became_active":"2024-09-21T02:04:33.626935+0000","last_became_peered":"2024-09-21T02:04:33.626935+0000","last_unstale":"2024-09-21T02:06:52.429092+0000","last_undegraded":"2024-09-21T02:06:52.429092+0000","last_fullsized":"2024-09-21T02:06:52.429092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:22.323730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.752288+0000","last_change":"2024-09-21T02:04:31.624697+0000","last_active":"2024-09-21T02:05:54.752288+0000","last_peered":"2024-09-21T02:05:54.752288+0000","last_clean":"2024-09-21T02:05:54.752288+0000","last_became_active":"2024-09-21T02:04:31.624309+0000","last_became_peered":"2024-09-21T02:04:31.624309+0000","last_unstale":"2024-09-21T02:05:54.752288+0000","last_undegraded":"2024-09-21T02:05:54.752288+0000","last_fullsized":"2024-09-21T02:05:54.752288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:57:40.712798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.842081+0000","last_change":"2024-09-21T02:05:54.841426+0000","last_active":"2024-09-21T02:05:54.842081+0000","last_peered":"2024-09-21T02:05:54.842081+0000","last_clean":"2024-09-21T02:05:54.842081+0000","last_became_active":"2024-09-21T02:05:54.840855+0000","last_became_peered":"2024-09-21T02:05:54.840855+0000","last_unstale":"2024-09-21T02:05:54.842081+0000","last_undegraded":"2024-09-21T02:05:54.842081+0000","last_fullsized":"2024-09-21T02:05:54.842081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:08:40.637427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438022+0000","last_change":"2024-09-21T02:04:33.629918+0000","last_active":"2024-09-21T02:06:52.438022+0000","last_peered":"2024-09-21T02:06:52.438022+0000","last_clean":"2024-09-21T02:06:52.438022+0000","last_became_active":"2024-09-21T02:04:33.629552+0000","last_became_peered":"2024-09-21T02:04:33.629552+0000","last_unstale":"2024-09-21T02:06:52.438022+0000","last_undegraded":"2024-09-21T02:06:52.438022+0000","last_fullsized":"2024-09-21T02:06:52.438022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:03:41.876590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.308608+0000","last_change":"2024-09-21T02:04:31.615378+0000","last_active":"2024-09-21T02:05:54.308608+0000","last_peered":"2024-09-21T02:05:54.308608+0000","last_clean":"2024-09-21T02:05:54.308608+0000","last_became_active":"2024-09-21T02:04:31.615042+0000","last_became_peered":"2024-09-21T02:04:31.615042+0000","last_unstale":"2024-09-21T02:05:54.308608+0000","last_undegraded":"2024-09-21T02:05:54.308608+0000","last_fullsized":"2024-09-21T02:05:54.308608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:50.450855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309526+0000","last_change":"2024-09-21T02:04:31.624357+0000","last_active":"2024-09-21T02:05:54.309526+0000","last_peered":"2024-09-21T02:05:54.309526+0000","last_clean":"2024-09-21T02:05:54.309526+0000","last_became_active":"2024-09-21T02:04:31.623970+0000","last_became_peered":"2024-09-21T02:04:31.623970+0000","last_unstale":"2024-09-21T02:05:54.309526+0000","last_undegraded":"2024-09-21T02:05:54.309526+0000","last_fullsized":"2024-09-21T02:05:54.309526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:30:01.519471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437429+0000","last_change":"2024-09-21T02:04:54.142456+0000","last_active":"2024-09-21T02:06:52.437429+0000","last_peered":"2024-09-21T02:06:52.437429+0000","last_clean":"2024-09-21T02:06:52.437429+0000","last_became_active":"2024-09-21T02:04:54.142066+0000","last_became_peered":"2024-09-21T02:04:54.142066+0000","last_unstale":"2024-09-21T02:06:52.437429+0000","last_undegraded":"2024-09-21T02:06:52.437429+0000","last_fullsized":"2024-09-21T02:06:52.437429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:51:39.896680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309835+0000","last_change":"2024-09-21T02:04:40.695925+0000","last_active":"2024-09-21T02:05:54.309835+0000","last_peered":"2024-09-21T02:05:54.309835+0000","last_clean":"2024-09-21T02:05:54.309835+0000","last_became_active":"2024-09-21T02:04:40.695457+0000","last_became_peered":"2024-09-21T02:04:40.695457+0000","last_unstale":"2024-09-21T02:05:54.309835+0000","last_undegraded":"2024-09-21T02:05:54.309835+0000","last_fullsized":"2024-09-21T02:05:54.309835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:20:34.366046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'8","reported_seq":167,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:07:12.911697+0000","last_change":"2024-09-21T02:04:31.621205+0000","last_active":"2024-09-21T02:07:12.911697+0000","last_peered":"2024-09-21T02:07:12.911697+0000","last_clean":"2024-09-21T02:07:12.911697+0000","last_became_active":"2024-09-21T02:04:31.620790+0000","last_became_peered":"2024-09-21T02:04:31.620790+0000","last_unstale":"2024-09-21T02:07:12.911697+0000","last_undegraded":"2024-09-21T02:07:12.911697+0000","last_fullsized":"2024-09-21T02:07:12.911697+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:30.598580+0000","last_clean_scrub_stamp":"2024-09-21T02:04:30.598580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:53.007959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.420899+0000","last_change":"2024-09-21T02:04:33.639404+0000","last_active":"2024-09-21T02:06:52.420899+0000","last_peered":"2024-09-21T02:06:52.420899+0000","last_clean":"2024-09-21T02:06:52.420899+0000","last_became_active":"2024-09-21T02:04:33.639029+0000","last_became_peered":"2024-09-21T02:04:33.639029+0000","last_unstale":"2024-09-21T02:06:52.420899+0000","last_undegraded":"2024-09-21T02:06:52.420899+0000","last_fullsized":"2024-09-21T02:06:52.420899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:30:52.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:05:54.309917+0000","last_change":"2024-09-21T02:04:40.690315+0000","last_active":"2024-09-21T02:05:54.309917+0000","last_peered":"2024-09-21T02:05:54.309917+0000","last_clean":"2024-09-21T02:05:54.309917+0000","last_became_active":"2024-09-21T02:04:40.689762+0000","last_became_peered":"2024-09-21T02:04:40.689762+0000","last_unstale":"2024-09-21T02:05:54.309917+0000","last_undegraded":"2024-09-21T02:05:54.309917+0000","last_fullsized":"2024-09-21T02:05:54.309917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:39.669191+0000","last_clean_scrub_stamp":"2024-09-21T02:04:39.669191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:36:06.007825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437860+0000","last_change":"2024-09-21T02:04:33.626456+0000","last_active":"2024-09-21T02:06:52.437860+0000","last_peered":"2024-09-21T02:06:52.437860+0000","last_clean":"2024-09-21T02:06:52.437860+0000","last_became_active":"2024-09-21T02:04:33.625454+0000","last_became_peered":"2024-09-21T02:04:33.625454+0000","last_unstale":"2024-09-21T02:06:52.437860+0000","last_undegraded":"2024-09-21T02:06:52.437860+0000","last_fullsized":"2024-09-21T02:06:52.437860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:11:18.781688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435988+0000","last_change":"2024-09-21T02:04:33.629869+0000","last_active":"2024-09-21T02:06:52.435988+0000","last_peered":"2024-09-21T02:06:52.435988+0000","last_clean":"2024-09-21T02:06:52.435988+0000","last_became_active":"2024-09-21T02:04:33.629184+0000","last_became_peered":"2024-09-21T02:04:33.629184+0000","last_unstale":"2024-09-21T02:06:52.435988+0000","last_undegraded":"2024-09-21T02:06:52.435988+0000","last_fullsized":"2024-09-21T02:06:52.435988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:26:13.818129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427485+0000","last_change":"2024-09-21T02:04:33.627193+0000","last_active":"2024-09-21T02:06:52.427485+0000","last_peered":"2024-09-21T02:06:52.427485+0000","last_clean":"2024-09-21T02:06:52.427485+0000","last_became_active":"2024-09-21T02:04:33.626401+0000","last_became_peered":"2024-09-21T02:04:33.626401+0000","last_unstale":"2024-09-21T02:06:52.427485+0000","last_undegraded":"2024-09-21T02:06:52.427485+0000","last_fullsized":"2024-09-21T02:06:52.427485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:27:34.081680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435191+0000","last_change":"2024-09-21T02:04:33.635507+0000","last_active":"2024-09-21T02:06:52.435191+0000","last_peered":"2024-09-21T02:06:52.435191+0000","last_clean":"2024-09-21T02:06:52.435191+0000","last_became_active":"2024-09-21T02:04:33.635159+0000","last_became_peered":"2024-09-21T02:04:33.635159+0000","last_unstale":"2024-09-21T02:06:52.435191+0000","last_undegraded":"2024-09-21T02:06:52.435191+0000","last_fullsized":"2024-09-21T02:06:52.435191+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:56:06.341478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431472+0000","last_change":"2024-09-21T02:04:33.629377+0000","last_active":"2024-09-21T02:06:52.431472+0000","last_peered":"2024-09-21T02:06:52.431472+0000","last_clean":"2024-09-21T02:06:52.431472+0000","last_became_active":"2024-09-21T02:04:33.628396+0000","last_became_peered":"2024-09-21T02:04:33.628396+0000","last_unstale":"2024-09-21T02:06:52.431472+0000","last_undegraded":"2024-09-21T02:06:52.431472+0000","last_fullsized":"2024-09-21T02:06:52.431472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:12:29.393234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436873+0000","last_change":"2024-09-21T02:04:33.632828+0000","last_active":"2024-09-21T02:06:52.436873+0000","last_peered":"2024-09-21T02:06:52.436873+0000","last_clean":"2024-09-21T02:06:52.436873+0000","last_became_active":"2024-09-21T02:04:33.632410+0000","last_became_peered":"2024-09-21T02:04:33.632410+0000","last_unstale":"2024-09-21T02:06:52.436873+0000","last_undegraded":"2024-09-21T02:06:52.436873+0000","last_fullsized":"2024-09-21T02:06:52.436873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:13:40.458621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.421562+0000","last_change":"2024-09-21T02:04:33.629294+0000","last_active":"2024-09-21T02:06:52.421562+0000","last_peered":"2024-09-21T02:06:52.421562+0000","last_clean":"2024-09-21T02:06:52.421562+0000","last_became_active":"2024-09-21T02:04:33.627498+0000","last_became_peered":"2024-09-21T02:04:33.627498+0000","last_unstale":"2024-09-21T02:06:52.421562+0000","last_undegraded":"2024-09-21T02:06:52.421562+0000","last_fullsized":"2024-09-21T02:06:52.421562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:16:20.514752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.429068+0000","last_change":"2024-09-21T02:04:33.638698+0000","last_active":"2024-09-21T02:06:52.429068+0000","last_peered":"2024-09-21T02:06:52.429068+0000","last_clean":"2024-09-21T02:06:52.429068+0000","last_became_active":"2024-09-21T02:04:33.638250+0000","last_became_peered":"2024-09-21T02:04:33.638250+0000","last_unstale":"2024-09-21T02:06:52.429068+0000","last_undegraded":"2024-09-21T02:06:52.429068+0000","last_fullsized":"2024-09-21T02:06:52.429068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:29.250823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435335+0000","last_change":"2024-09-21T02:04:33.634071+0000","last_active":"2024-09-21T02:06:52.435335+0000","last_peered":"2024-09-21T02:06:52.435335+0000","last_clean":"2024-09-21T02:06:52.435335+0000","last_became_active":"2024-09-21T02:04:33.633177+0000","last_became_peered":"2024-09-21T02:04:33.633177+0000","last_unstale":"2024-09-21T02:06:52.435335+0000","last_undegraded":"2024-09-21T02:06:52.435335+0000","last_fullsized":"2024-09-21T02:06:52.435335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:36:03.736306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438080+0000","last_change":"2024-09-21T02:04:33.631560+0000","last_active":"2024-09-21T02:06:52.438080+0000","last_peered":"2024-09-21T02:06:52.438080+0000","last_clean":"2024-09-21T02:06:52.438080+0000","last_became_active":"2024-09-21T02:04:33.631189+0000","last_became_peered":"2024-09-21T02:04:33.631189+0000","last_unstale":"2024-09-21T02:06:52.438080+0000","last_undegraded":"2024-09-21T02:06:52.438080+0000","last_fullsized":"2024-09-21T02:06:52.438080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:05:48.142004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447599+0000","last_change":"2024-09-21T02:04:33.635870+0000","last_active":"2024-09-21T02:06:52.447599+0000","last_peered":"2024-09-21T02:06:52.447599+0000","last_clean":"2024-09-21T02:06:52.447599+0000","last_became_active":"2024-09-21T02:04:33.635481+0000","last_became_peered":"2024-09-21T02:04:33.635481+0000","last_unstale":"2024-09-21T02:06:52.447599+0000","last_undegraded":"2024-09-21T02:06:52.447599+0000","last_fullsized":"2024-09-21T02:06:52.447599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:54:00.012100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.446393+0000","last_change":"2024-09-21T02:04:33.630322+0000","last_active":"2024-09-21T02:06:52.446393+0000","last_peered":"2024-09-21T02:06:52.446393+0000","last_clean":"2024-09-21T02:06:52.446393+0000","last_became_active":"2024-09-21T02:04:33.629910+0000","last_became_peered":"2024-09-21T02:04:33.629910+0000","last_unstale":"2024-09-21T02:06:52.446393+0000","last_undegraded":"2024-09-21T02:06:52.446393+0000","last_fullsized":"2024-09-21T02:06:52.446393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:13.685015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436021+0000","last_change":"2024-09-21T02:04:33.637254+0000","last_active":"2024-09-21T02:06:52.436021+0000","last_peered":"2024-09-21T02:06:52.436021+0000","last_clean":"2024-09-21T02:06:52.436021+0000","last_became_active":"2024-09-21T02:04:33.636892+0000","last_became_peered":"2024-09-21T02:04:33.636892+0000","last_unstale":"2024-09-21T02:06:52.436021+0000","last_undegraded":"2024-09-21T02:06:52.436021+0000","last_fullsized":"2024-09-21T02:06:52.436021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:42:09.582343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436853+0000","last_change":"2024-09-21T02:04:33.635144+0000","last_active":"2024-09-21T02:06:52.436853+0000","last_peered":"2024-09-21T02:06:52.436853+0000","last_clean":"2024-09-21T02:06:52.436853+0000","last_became_active":"2024-09-21T02:04:33.634738+0000","last_became_peered":"2024-09-21T02:04:33.634738+0000","last_unstale":"2024-09-21T02:06:52.436853+0000","last_undegraded":"2024-09-21T02:06:52.436853+0000","last_fullsized":"2024-09-21T02:06:52.436853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:43.840157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.445987+0000","last_change":"2024-09-21T02:04:33.626061+0000","last_active":"2024-09-21T02:06:52.445987+0000","last_peered":"2024-09-21T02:06:52.445987+0000","last_clean":"2024-09-21T02:06:52.445987+0000","last_became_active":"2024-09-21T02:04:33.625702+0000","last_became_peered":"2024-09-21T02:04:33.625702+0000","last_unstale":"2024-09-21T02:06:52.445987+0000","last_undegraded":"2024-09-21T02:06:52.445987+0000","last_fullsized":"2024-09-21T02:06:52.445987+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:46.061325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434828+0000","last_change":"2024-09-21T02:04:33.635383+0000","last_active":"2024-09-21T02:06:52.434828+0000","last_peered":"2024-09-21T02:06:52.434828+0000","last_clean":"2024-09-21T02:06:52.434828+0000","last_became_active":"2024-09-21T02:04:33.634768+0000","last_became_peered":"2024-09-21T02:04:33.634768+0000","last_unstale":"2024-09-21T02:06:52.434828+0000","last_undegraded":"2024-09-21T02:06:52.434828+0000","last_fullsized":"2024-09-21T02:06:52.434828+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:51.263159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.447148+0000","last_change":"2024-09-21T02:04:33.636779+0000","last_active":"2024-09-21T02:06:52.447148+0000","last_peered":"2024-09-21T02:06:52.447148+0000","last_clean":"2024-09-21T02:06:52.447148+0000","last_became_active":"2024-09-21T02:04:33.636387+0000","last_became_peered":"2024-09-21T02:04:33.636387+0000","last_unstale":"2024-09-21T02:06:52.447148+0000","last_undegraded":"2024-09-21T02:06:52.447148+0000","last_fullsized":"2024-09-21T02:06:52.447148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:22.071643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432764+0000","last_change":"2024-09-21T02:04:33.639340+0000","last_active":"2024-09-21T02:06:52.432764+0000","last_peered":"2024-09-21T02:06:52.432764+0000","last_clean":"2024-09-21T02:06:52.432764+0000","last_became_active":"2024-09-21T02:04:33.637905+0000","last_became_peered":"2024-09-21T02:04:33.637905+0000","last_unstale":"2024-09-21T02:06:52.432764+0000","last_undegraded":"2024-09-21T02:06:52.432764+0000","last_fullsized":"2024-09-21T02:06:52.432764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:21.022761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.428308+0000","last_change":"2024-09-21T02:04:33.639513+0000","last_active":"2024-09-21T02:06:52.428308+0000","last_peered":"2024-09-21T02:06:52.428308+0000","last_clean":"2024-09-21T02:06:52.428308+0000","last_became_active":"2024-09-21T02:04:33.638452+0000","last_became_peered":"2024-09-21T02:04:33.638452+0000","last_unstale":"2024-09-21T02:06:52.428308+0000","last_undegraded":"2024-09-21T02:06:52.428308+0000","last_fullsized":"2024-09-21T02:06:52.428308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:36.152926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.437194+0000","last_change":"2024-09-21T02:04:33.629975+0000","last_active":"2024-09-21T02:06:52.437194+0000","last_peered":"2024-09-21T02:06:52.437194+0000","last_clean":"2024-09-21T02:06:52.437194+0000","last_became_active":"2024-09-21T02:04:33.629210+0000","last_became_peered":"2024-09-21T02:04:33.629210+0000","last_unstale":"2024-09-21T02:06:52.437194+0000","last_undegraded":"2024-09-21T02:06:52.437194+0000","last_fullsized":"2024-09-21T02:06:52.437194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:24.261233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430717+0000","last_change":"2024-09-21T02:04:33.631206+0000","last_active":"2024-09-21T02:06:52.430717+0000","last_peered":"2024-09-21T02:06:52.430717+0000","last_clean":"2024-09-21T02:06:52.430717+0000","last_became_active":"2024-09-21T02:04:33.630371+0000","last_became_peered":"2024-09-21T02:04:33.630371+0000","last_unstale":"2024-09-21T02:06:52.430717+0000","last_undegraded":"2024-09-21T02:06:52.430717+0000","last_fullsized":"2024-09-21T02:06:52.430717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:00:36.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.444213+0000","last_change":"2024-09-21T02:04:33.636299+0000","last_active":"2024-09-21T02:06:52.444213+0000","last_peered":"2024-09-21T02:06:52.444213+0000","last_clean":"2024-09-21T02:06:52.444213+0000","last_became_active":"2024-09-21T02:04:33.635593+0000","last_became_peered":"2024-09-21T02:04:33.635593+0000","last_unstale":"2024-09-21T02:06:52.444213+0000","last_undegraded":"2024-09-21T02:06:52.444213+0000","last_fullsized":"2024-09-21T02:06:52.444213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:37.379603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.432020+0000","last_change":"2024-09-21T02:04:53.748428+0000","last_active":"2024-09-21T02:06:52.432020+0000","last_peered":"2024-09-21T02:06:52.432020+0000","last_clean":"2024-09-21T02:06:52.432020+0000","last_became_active":"2024-09-21T02:04:53.747922+0000","last_became_peered":"2024-09-21T02:04:53.747922+0000","last_unstale":"2024-09-21T02:06:52.432020+0000","last_undegraded":"2024-09-21T02:06:52.432020+0000","last_fullsized":"2024-09-21T02:06:52.432020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:50.646500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.438660+0000","last_change":"2024-09-21T02:04:33.634408+0000","last_active":"2024-09-21T02:06:52.438660+0000","last_peered":"2024-09-21T02:06:52.438660+0000","last_clean":"2024-09-21T02:06:52.438660+0000","last_became_active":"2024-09-21T02:04:33.633973+0000","last_became_peered":"2024-09-21T02:04:33.633973+0000","last_unstale":"2024-09-21T02:06:52.438660+0000","last_undegraded":"2024-09-21T02:06:52.438660+0000","last_fullsized":"2024-09-21T02:06:52.438660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:32.131036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.436322+0000","last_change":"2024-09-21T02:04:33.635576+0000","last_active":"2024-09-21T02:06:52.436322+0000","last_peered":"2024-09-21T02:06:52.436322+0000","last_clean":"2024-09-21T02:06:52.436322+0000","last_became_active":"2024-09-21T02:04:33.634984+0000","last_became_peered":"2024-09-21T02:04:33.634984+0000","last_unstale":"2024-09-21T02:06:52.436322+0000","last_undegraded":"2024-09-21T02:06:52.436322+0000","last_fullsized":"2024-09-21T02:06:52.436322+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:56:01.715383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.435393+0000","last_change":"2024-09-21T02:04:33.629670+0000","last_active":"2024-09-21T02:06:52.435393+0000","last_peered":"2024-09-21T02:06:52.435393+0000","last_clean":"2024-09-21T02:06:52.435393+0000","last_became_active":"2024-09-21T02:04:33.629326+0000","last_became_peered":"2024-09-21T02:04:33.629326+0000","last_unstale":"2024-09-21T02:06:52.435393+0000","last_undegraded":"2024-09-21T02:06:52.435393+0000","last_fullsized":"2024-09-21T02:06:52.435393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:29:00.505521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.426787+0000","last_change":"2024-09-21T02:04:33.628239+0000","last_active":"2024-09-21T02:06:52.426787+0000","last_peered":"2024-09-21T02:06:52.426787+0000","last_clean":"2024-09-21T02:06:52.426787+0000","last_became_active":"2024-09-21T02:04:33.627876+0000","last_became_peered":"2024-09-21T02:04:33.627876+0000","last_unstale":"2024-09-21T02:06:52.426787+0000","last_undegraded":"2024-09-21T02:06:52.426787+0000","last_fullsized":"2024-09-21T02:06:52.426787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:16:12.500451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.434645+0000","last_change":"2024-09-21T02:04:33.630851+0000","last_active":"2024-09-21T02:06:52.434645+0000","last_peered":"2024-09-21T02:06:52.434645+0000","last_clean":"2024-09-21T02:06:52.434645+0000","last_became_active":"2024-09-21T02:04:33.630414+0000","last_became_peered":"2024-09-21T02:04:33.630414+0000","last_unstale":"2024-09-21T02:06:52.434645+0000","last_undegraded":"2024-09-21T02:06:52.434645+0000","last_fullsized":"2024-09-21T02:06:52.434645+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:11:17.465846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.430839+0000","last_change":"2024-09-21T02:04:33.636946+0000","last_active":"2024-09-21T02:06:52.430839+0000","last_peered":"2024-09-21T02:06:52.430839+0000","last_clean":"2024-09-21T02:06:52.430839+0000","last_became_active":"2024-09-21T02:04:33.636606+0000","last_became_peered":"2024-09-21T02:04:33.636606+0000","last_unstale":"2024-09-21T02:06:52.430839+0000","last_undegraded":"2024-09-21T02:06:52.430839+0000","last_fullsized":"2024-09-21T02:06:52.430839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:31:51.729741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.433464+0000","last_change":"2024-09-21T02:04:33.631823+0000","last_active":"2024-09-21T02:06:52.433464+0000","last_peered":"2024-09-21T02:06:52.433464+0000","last_clean":"2024-09-21T02:06:52.433464+0000","last_became_active":"2024-09-21T02:04:33.631439+0000","last_became_peered":"2024-09-21T02:04:33.631439+0000","last_unstale":"2024-09-21T02:06:52.433464+0000","last_undegraded":"2024-09-21T02:06:52.433464+0000","last_fullsized":"2024-09-21T02:06:52.433464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:09:11.807400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":39,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.427717+0000","last_change":"2024-09-21T02:04:33.641046+0000","last_active":"2024-09-21T02:06:52.427717+0000","last_peered":"2024-09-21T02:06:52.427717+0000","last_clean":"2024-09-21T02:06:52.427717+0000","last_became_active":"2024-09-21T02:04:33.640276+0000","last_became_peered":"2024-09-21T02:04:33.640276+0000","last_unstale":"2024-09-21T02:06:52.427717+0000","last_undegraded":"2024-09-21T02:06:52.427717+0000","last_fullsized":"2024-09-21T02:06:52.427717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:17.230483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"86'8","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-21T02:06:52.431461+0000","last_change":"2024-09-21T02:04:33.632252+0000","last_active":"2024-09-21T02:06:52.431461+0000","last_peered":"2024-09-21T02:06:52.431461+0000","last_clean":"2024-09-21T02:06:52.431461+0000","last_became_active":"2024-09-21T02:04:33.631516+0000","last_became_peered":"2024-09-21T02:04:33.631516+0000","last_unstale":"2024-09-21T02:06:52.431461+0000","last_undegraded":"2024-09-21T02:06:52.431461+0000","last_fullsized":"2024-09-21T02:06:52.431461+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T02:04:32.604971+0000","last_clean_scrub_stamp":"2024-09-21T02:04:32.604971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:42:44.960253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":868817,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1510,"num_read_kb":978,"num_write":501,"num_write_kb":2260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":835584,"data_stored":2606451,"data_compressed":112140,"data_compressed_allocated":147456,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":249624,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710766,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":501553,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874290,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1163264,"data_stored":1553336,"data_compressed":84124,"data_compressed_allocated":110592,"data_compressed_original":1040384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037816,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":897658,"data_compressed":48968,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":893828,"data_compressed":48968,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1099905,"data_compressed":66033,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561095,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":496875,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757325,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1085440,"data_stored":1487213,"data_compressed":84124,"data_compressed_allocated":110592,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":500949,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1040384,"data_stored":1282550,"data_compressed":67059,"data_compressed_allocated":90112,"data_compressed_original":843776,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378716,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":565992,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542241,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":569320,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705766,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":498331,"data_compressed":29679,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34736,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36354,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":74192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":75213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":787422,"data_compressed":37380,"data_compressed_allocated":49152,"data_compressed_original":761856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397621,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":394604,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":467840,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T02:07:15.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:07:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:15.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:15 smithi123 ceph-mon[33453]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 59 op/s 2024-09-21T02:07:15.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:15 smithi123 ceph-mon[33453]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:15.618 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:15 smithi016 ceph-mon[31245]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 59 op/s 2024-09-21T02:07:15.618 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:15 smithi016 ceph-mon[31245]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:15.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:15 smithi161 ceph-mon[34061]: pgmap v355: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 59 op/s 2024-09-21T02:07:15.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:15 smithi161 ceph-mon[34061]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 -- 172.21.15.16:0/4254526062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7381062b0 msgr2=0x7fb7381066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 --2- 172.21.15.16:0/4254526062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7381062b0 0x7fb7381066c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb724009900 tx=0x7fb72402a410 comp rx=0 tx=0).stop 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 -- 172.21.15.16:0/4254526062 shutdown_connections 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 --2- 172.21.15.16:0/4254526062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb738106c00 0x7fb738110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 --2- 172.21.15.16:0/4254526062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7381062b0 0x7fb7381066c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 --2- 172.21.15.16:0/4254526062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb7381024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 -- 172.21.15.16:0/4254526062 >> 172.21.15.16:0/4254526062 conn(0x7fb738079e40 msgr2=0x7fb73807a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:16.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 -- 172.21.15.16:0/4254526062 shutdown_connections 2024-09-21T02:07:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 -- 172.21.15.16:0/4254526062 wait complete. 2024-09-21T02:07:16.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.056+0000 7fb73eadd640 1 Processor -- start 2024-09-21T02:07:16.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 -- start start 2024-09-21T02:07:16.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb738103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:16.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 0x7fb7381035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 0x7fb738103ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7381139b0 con 0x7fb738106c00 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb738113830 con 0x7fb7381062b0 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73eadd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb738113b30 con 0x7fb7381020c0 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73c879640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb738103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb737fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 0x7fb7381035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb73d07a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 0x7fb738103ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb737fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 0x7fb7381035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:53026/0 (socket says 172.21.15.16:53026) 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.057+0000 7fb737fff640 1 -- 172.21.15.16:0/2043360067 learned_addr learned my addr 172.21.15.16:0/2043360067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 -- 172.21.15.16:0/2043360067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 msgr2=0x7fb7381035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 0x7fb7381035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 -- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 msgr2=0x7fb738103ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 0x7fb738103ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 -- 172.21.15.16:0/2043360067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7381a8760 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73d07a640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 0x7fb738103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73c879640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb738103060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00d720 tx=0x7fb72c00dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c017070 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7381a8a50 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb72c00de30 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c004de0 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.058+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7381a8f60 con 0x7fb7381020c0 2024-09-21T02:07:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.059+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7381024d0 con 0x7fb7381020c0 2024-09-21T02:07:16.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.061+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fb72c0099d0 con 0x7fb7381020c0 2024-09-21T02:07:16.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.062+0000 7fb735ffb640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 0x7fb7280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:16.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.062+0000 7fb737fff640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 0x7fb7280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:16.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.062+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fb72c0e7920 con 0x7fb7381020c0 2024-09-21T02:07:16.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.062+0000 7fb737fff640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 0x7fb7280b8b00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb724000c00 tx=0x7fb7240058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:16.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.065+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb72c020020 con 0x7fb7381020c0 2024-09-21T02:07:16.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.295+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb73802e640 con 0x7fb7381020c0 2024-09-21T02:07:16.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.296+0000 7fb735ffb640 1 -- 172.21.15.16:0/2043360067 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fb72c0adbc0 con 0x7fb7381020c0 2024-09-21T02:07:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:16.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:07:16.298 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:07:16.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.298+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 msgr2=0x7fb7280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.298+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 0x7fb7280b8b00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb724000c00 tx=0x7fb7240058d0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.298+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 msgr2=0x7fb738103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.298+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb738103060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00d720 tx=0x7fb72c00dbe0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 shutdown_connections 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb7280b65f0 0x7fb7280b8b00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb738106c00 0x7fb738103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7381062b0 0x7fb7381035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 --2- 172.21.15.16:0/2043360067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7381020c0 0x7fb738103060 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 >> 172.21.15.16:0/2043360067 conn(0x7fb738079e40 msgr2=0x7fb738107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 shutdown_connections 2024-09-21T02:07:16.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:16.299+0000 7fb73eadd640 1 -- 172.21.15.16:0/2043360067 wait complete. 2024-09-21T02:07:17.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:07:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:17.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:17 smithi016 ceph-mon[31245]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:17.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2043360067' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:17 smithi123 ceph-mon[33453]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:17 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2043360067' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:17.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:17 smithi161 ceph-mon[34061]: pgmap v356: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:17.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:17 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2043360067' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 -- 172.21.15.16:0/3580354596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94103b80 msgr2=0x7f7e94103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3580354596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94103b80 0x7f7e94103f90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009900 tx=0x7f7e8402a410 comp rx=0 tx=0).stop 2024-09-21T02:07:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 -- 172.21.15.16:0/3580354596 shutdown_connections 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3580354596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94109e10 0x7f7e94110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3580354596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 0x7f7e941098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3580354596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94103b80 0x7f7e94103f90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 -- 172.21.15.16:0/3580354596 >> 172.21.15.16:0/3580354596 conn(0x7f7e940ff610 msgr2=0x7f7e94101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 -- 172.21.15.16:0/3580354596 shutdown_connections 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.766+0000 7f7e9a625640 1 -- 172.21.15.16:0/3580354596 wait complete. 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 Processor -- start 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 -- start start 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94103b80 0x7f7e9419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 0x7f7e9419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e941139d0 con 0x7f7e94109e10 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7e94113850 con 0x7f7e94103b80 2024-09-21T02:07:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e9a625640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7e94113b50 con 0x7f7e941094c0 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e98bc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e98bc2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46200/0 (socket says 172.21.15.16:46200) 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e98bc2640 1 -- 172.21.15.16:0/3852179294 learned_addr learned my addr 172.21.15.16:0/3852179294 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e93fff640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94103b80 0x7f7e9419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.767+0000 7f7e937fe640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 0x7f7e9419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 -- 172.21.15.16:0/3852179294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 msgr2=0x7f7e9419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 0x7f7e9419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 -- 172.21.15.16:0/3852179294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94103b80 msgr2=0x7f7e9419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94103b80 0x7f7e9419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 -- 172.21.15.16:0/3852179294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e941a0350 con 0x7f7e94109e10 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e98bc2640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00eaa0 tx=0x7f7e8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00cdb0 con 0x7f7e94109e10 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e8c005020 con 0x7f7e94109e10 2024-09-21T02:07:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0116d0 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e941a0640 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.768+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e941a6da0 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.769+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e94103f90 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.771+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f7e8c0051d0 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.771+0000 7f7e917fa640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 0x7f7e7c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.772+0000 7f7e93fff640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 0x7f7e7c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.772+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f7e8c0e7e40 con 0x7f7e94109e10 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.772+0000 7f7e93fff640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 0x7f7e7c0b8b80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006010 tx=0x7f7e84000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:17.775+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7e8c016030 con 0x7f7e94109e10 2024-09-21T02:07:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.002+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e9419b510 con 0x7f7e94109e10 2024-09-21T02:07:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.003+0000 7f7e917fa640 1 -- 172.21.15.16:0/3852179294 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f7e9419b510 con 0x7f7e94109e10 2024-09-21T02:07:18.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:18.004 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:07:18.004 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:07:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.005+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 msgr2=0x7f7e7c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.005+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 0x7f7e7c0b8b80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006010 tx=0x7f7e84000bc0 comp rx=0 tx=0).stop 2024-09-21T02:07:18.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.005+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 msgr2=0x7f7e9419fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.005+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00eaa0 tx=0x7f7e8c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 shutdown_connections 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e7c0b6670 0x7f7e7c0b8b80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e94109e10 0x7f7e9419fc20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e941094c0 0x7f7e9419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 --2- 172.21.15.16:0/3852179294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e94103b80 0x7f7e9419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 >> 172.21.15.16:0/3852179294 conn(0x7f7e940ff610 msgr2=0x7f7e94101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 shutdown_connections 2024-09-21T02:07:18.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:18.006+0000 7f7e9a625640 1 -- 172.21.15.16:0/3852179294 wait complete. 2024-09-21T02:07:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:18 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3852179294' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:18.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:18 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3852179294' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:18.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:18 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3852179294' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:18.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:07:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:19.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:19 smithi016 ceph-mon[31245]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:19.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:19 smithi123 ceph-mon[33453]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- 172.21.15.16:0/1341531884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c103b90 msgr2=0x7ff26c103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 --2- 172.21.15.16:0/1341531884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c103b90 0x7ff26c103fa0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009900 tx=0x7ff25c02a410 comp rx=0 tx=0).stop 2024-09-21T02:07:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- 172.21.15.16:0/1341531884 shutdown_connections 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 --2- 172.21.15.16:0/1341531884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c109e20 0x7ff26c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 --2- 172.21.15.16:0/1341531884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c1094d0 0x7ff26c1098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 --2- 172.21.15.16:0/1341531884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c103b90 0x7ff26c103fa0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- 172.21.15.16:0/1341531884 >> 172.21.15.16:0/1341531884 conn(0x7ff26c0ff600 msgr2=0x7ff26c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- 172.21.15.16:0/1341531884 shutdown_connections 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- 172.21.15.16:0/1341531884 wait complete. 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 Processor -- start 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.522+0000 7ff271e5c640 1 -- start start 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 0x7ff26c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c1094d0 0x7ff26c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1139e0 con 0x7ff26c109e20 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c113860 con 0x7ff26c103b90 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff271e5c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c113b60 con 0x7ff26c1094d0 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46222/0 (socket says 172.21.15.16:46222) 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 -- 172.21.15.16:0/3531068798 learned_addr learned my addr 172.21.15.16:0/3531068798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26b7fe640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 0x7ff26c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff263fff640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c1094d0 0x7ff26c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 -- 172.21.15.16:0/3531068798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c1094d0 msgr2=0x7ff26c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c1094d0 0x7ff26c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 -- 172.21.15.16:0/3531068798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 msgr2=0x7ff26c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 0x7ff26c19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 -- 172.21.15.16:0/3531068798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c1a2620 con 0x7ff26c109e20 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26b7fe640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 0x7ff26c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.523+0000 7ff26bfff640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff264009fd0 tx=0x7ff26400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.524+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264017070 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.524+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff264003e00 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.524+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26c1a2910 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.524+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264010460 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.524+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26c1a9070 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.525+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26c104e10 con 0x7ff26c109e20 2024-09-21T02:07:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.526+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7ff264003970 con 0x7ff26c109e20 2024-09-21T02:07:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.527+0000 7ff2697fa640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 0x7ff2580b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.528+0000 7ff26b7fe640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 0x7ff2580b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.528+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff2640e7b80 con 0x7ff26c109e20 2024-09-21T02:07:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.528+0000 7ff26b7fe640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 0x7ff2580b8b00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff25c006010 tx=0x7ff25c005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.531+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff2640addc0 con 0x7ff26c109e20 2024-09-21T02:07:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:19 smithi161 ceph-mon[34061]: pgmap v357: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-09-21T02:07:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.719+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff26c103fa0 con 0x7ff26c109e20 2024-09-21T02:07:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.721+0000 7ff2697fa640 1 -- 172.21.15.16:0/3531068798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+25198 (secure 0 0 0) 0x7ff2640b21a0 con 0x7ff26c109e20 2024-09-21T02:07:19.722 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:19.722 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:05:54.299820+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 msgr2=0x7ff2580b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 0x7ff2580b8b00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff25c006010 tx=0x7ff25c005a20 comp rx=0 tx=0).stop 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 msgr2=0x7ff26c1a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff264009fd0 tx=0x7ff26400b260 comp rx=0 tx=0).stop 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 shutdown_connections 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff2580b65f0 0x7ff2580b8b00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff26c109e20 0x7ff26c1a1ef0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff26c1094d0 0x7ff26c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 --2- 172.21.15.16:0/3531068798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff26c103b90 0x7ff26c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 >> 172.21.15.16:0/3531068798 conn(0x7ff26c0ff600 msgr2=0x7ff26c108bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 shutdown_connections 2024-09-21T02:07:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:19.723+0000 7ff271e5c640 1 -- 172.21.15.16:0/3531068798 wait complete. 2024-09-21T02:07:20.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3531068798' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:07:20.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mon.a --format=json config get run_dir 2024-09-21T02:07:20.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3531068798' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:07:20.651 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:20.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3531068798' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:07:21.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:21 smithi016 ceph-mon[31245]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T02:07:21.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:21 smithi123 ceph-mon[33453]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T02:07:21.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:21 smithi161 ceph-mon[34061]: pgmap v358: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-09-21T02:07:21.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"run_dir":"/var/run/ceph"} 2024-09-21T02:07:21.889 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-21T02:07:21.889 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 7864 2024-09-21T02:07:21.889 DEBUG:tasks.quiescer:Yielding 2024-09-21T02:07:21.889 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T02:07:21.904 INFO:tasks.workunit:Pulling workunits from ref 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:07:21.905 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T02:07:21.905 INFO:tasks.workunit:timeout=3h 2024-09-21T02:07:21.905 INFO:tasks.workunit:cleanup=True 2024-09-21T02:07:21.905 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:21.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6ch/108d Inode: 1 Links: 2 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-21 02:05:24.313234420 +0000 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-21 02:05:24.313234420 +0000 2024-09-21T02:07:21.924 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-21 02:07:07.474222849 +0000 2024-09-21T02:07:21.925 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-21T02:07:21.925 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-21T02:07:21.925 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-21T02:07:21.997 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T02:07:22.053 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T02:07:22.173 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 -- 172.21.15.16:0/2362588437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 msgr2=0x7f29a0110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 --2- 172.21.15.16:0/2362588437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a0110640 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2998009ca0 tx=0x7f299802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 -- 172.21.15.16:0/2362588437 shutdown_connections 2024-09-21T02:07:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 --2- 172.21.15.16:0/2362588437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a0110640 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 --2- 172.21.15.16:0/2362588437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 0x7f29a01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 --2- 172.21.15.16:0/2362588437 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a0103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 -- 172.21.15.16:0/2362588437 >> 172.21.15.16:0/2362588437 conn(0x7f29a00ff5d0 msgr2=0x7f29a0101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 -- 172.21.15.16:0/2362588437 shutdown_connections 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.673+0000 7f29a784e640 1 -- 172.21.15.16:0/2362588437 wait complete. 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 Processor -- start 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 -- start start 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 0x7f29a019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:22.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29a01139d0 con 0x7f29a0109e10 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0113850 con 0x7f29a01094c0 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a784e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f29a0113b50 con 0x7f29a0103b80 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a5deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a5deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46246/0 (socket says 172.21.15.16:46246) 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a5deb640 1 -- 172.21.15.16:0/3558144889 learned_addr learned my addr 172.21.15.16:0/3558144889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a4de9640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 0x7f29a019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.674+0000 7f29a55ea640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 -- 172.21.15.16:0/3558144889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 msgr2=0x7f29a019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 -- 172.21.15.16:0/3558144889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 msgr2=0x7f29a019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 0x7f29a019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 -- 172.21.15.16:0/3558144889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a01a0230 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a55ea640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a5deb640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2998005e30 tx=0x7f29980093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998032d10 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a01a04c0 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2998032eb0 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998030a60 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.675+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a01a0a00 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.676+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a0103f90 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.678+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f2998030c00 con 0x7f29a0109e10 2024-09-21T02:07:22.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.679+0000 7f29967fc640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 0x7f298c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.679+0000 7f29a55ea640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 0x7f298c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.679+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2998108b50 con 0x7f29a0109e10 2024-09-21T02:07:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.679+0000 7f29a55ea640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 0x7f298c0b8b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f299c009c20 tx=0x7f299c005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:22.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.683+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f29980cedf0 con 0x7f29a0109e10 2024-09-21T02:07:22.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.913+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29a002e640 con 0x7f29a0109e10 2024-09-21T02:07:22.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.914+0000 7f29967fc640 1 -- 172.21.15.16:0/3558144889 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f29980d31d0 con 0x7f29a0109e10 2024-09-21T02:07:22.915 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:22.915 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:07:22.915 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 msgr2=0x7f298c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 0x7f298c0b8b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f299c009c20 tx=0x7f299c005f90 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 msgr2=0x7f29a019fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2998005e30 tx=0x7f29980093d0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 shutdown_connections 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f298c0b6620 0x7f298c0b8b30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29a0109e10 0x7f29a019fbc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a01094c0 0x7f29a019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 --2- 172.21.15.16:0/3558144889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f29a0103b80 0x7f29a019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 >> 172.21.15.16:0/3558144889 conn(0x7f29a00ff5d0 msgr2=0x7f29a01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 shutdown_connections 2024-09-21T02:07:22.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:22.917+0000 7f29a784e640 1 -- 172.21.15.16:0/3558144889 wait complete. 2024-09-21T02:07:23.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:23 smithi123 ceph-mon[33453]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 113 KiB/s wr, 49 op/s 2024-09-21T02:07:23.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:23 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3558144889' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:23.536 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:23 smithi016 ceph-mon[31245]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 113 KiB/s wr, 49 op/s 2024-09-21T02:07:23.536 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:23 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3558144889' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:23.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:23 smithi161 ceph-mon[34061]: pgmap v359: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 113 KiB/s wr, 49 op/s 2024-09-21T02:07:23.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:23 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3558144889' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:23.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-21T02:04:34.359195+0000', 'modified': '2024-09-21T02:07:07.873572+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24710, 'mds_1': 24683, 'mds_2': 24667, 'mds_3': 24659, 'mds_4': 24655}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14841': {'gid': 14841, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6839/1623043260', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1623043260}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1623043260}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14853': {'gid': 14853, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6841/3323661058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6840', 'nonce': 3323661058}, {'type': 'v1', 'addr': '172.21.15.16:6841', 'nonce': 3323661058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24595': {'gid': 24595, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6833/2556025136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6832', 'nonce': 2556025136}, {'type': 'v1', 'addr': '172.21.15.161:6833', 'nonce': 2556025136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24626': {'gid': 24626, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.123:6833/3641506877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6832', 'nonce': 3641506877}, {'type': 'v1', 'addr': '172.21.15.123:6833', 'nonce': 3641506877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24643': {'gid': 24643, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/1279942273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 1279942273}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 1279942273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24655': {'gid': 24655, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.161:6837/2373808610', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 2373808610}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 2373808610}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24659': {'gid': 24659, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.123:6835/2421556237', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2421556237}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2421556237}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24667': {'gid': 24667, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.161:6839/2696881795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2696881795}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2696881795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24683': {'gid': 24683, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.123:6837/1023560595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6836', 'nonce': 1023560595}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 1023560595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24710': {'gid': 24710, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6839/1116723582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 1116723582}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 1116723582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24710, 'qdb_cluster': [24710, 24683, 24667, 24659, 24655]} 2024-09-21T02:07:23.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-21T02:07:23.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.c status 2024-09-21T02:07:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 -- 172.21.15.16:0/1926442199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c108d20 msgr2=0x7fe27c10f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 --2- 172.21.15.16:0/1926442199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c108d20 0x7fe27c10f550 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe274009ca0 tx=0x7fe27402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 -- 172.21.15.16:0/1926442199 shutdown_connections 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 --2- 172.21.15.16:0/1926442199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c108d20 0x7fe27c10f550 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 --2- 172.21.15.16:0/1926442199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c1083d0 0x7fe27c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 --2- 172.21.15.16:0/1926442199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 0x7fe27c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 -- 172.21.15.16:0/1926442199 >> 172.21.15.16:0/1926442199 conn(0x7fe27c079760 msgr2=0x7fe27c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 -- 172.21.15.16:0/1926442199 shutdown_connections 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.560+0000 7fe284528640 1 -- 172.21.15.16:0/1926442199 wait complete. 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 Processor -- start 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 -- start start 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 0x7fe27c19b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c108d20 0x7fe27c19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1128f0 con 0x7fe27c1083d0 2024-09-21T02:07:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c112770 con 0x7fe27c108d20 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe284528640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c112a70 con 0x7fe27c102ae0 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46256/0 (socket says 172.21.15.16:46256) 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 -- 172.21.15.16:0/3859355891 learned_addr learned my addr 172.21.15.16:0/3859355891 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c108d20 0x7fe27c19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe2822c4640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 0x7fe27c19b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 -- 172.21.15.16:0/3859355891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 msgr2=0x7fe27c19b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 0x7fe27c19b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 -- 172.21.15.16:0/3859355891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c108d20 msgr2=0x7fe27c19bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c108d20 0x7fe27c19bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.561+0000 7fe281ac3640 1 -- 172.21.15.16:0/3859355891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c1a6450 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe27000b450 tx=0x7fe27000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe26f7fe640 1 -- 172.21.15.16:0/3859355891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270015020 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe26f7fe640 1 -- 172.21.15.16:0/3859355891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe270011d20 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27c1a6740 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe26f7fe640 1 -- 172.21.15.16:0/3859355891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270010e10 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.562+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe27c1a6c80 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.564+0000 7fe28aa9c740 1 Processor -- start 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 -- start start 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe26f7fe640 1 -- 172.21.15.16:0/3859355891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fe270018020 con 0x7fe27c1083d0 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0decf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a1e0a0 0x5562a0ded440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 0x5562a0ded980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562a0d64c10 con 0x5562a0a26280 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5562a0d54040 con 0x5562a0a46f60 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5562a0d57e50 con 0x5562a0a1e0a0 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 0x5562a0ded980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 0x5562a0ded980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46272/0 (socket says 172.21.15.16:46272) 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe2822c4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a1e0a0 0x5562a0ded440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe281ac3640 1 -- 172.21.15.16:0/3203852438 learned_addr learned my addr 172.21.15.16:0/3203852438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.565+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0decf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 -- 172.21.15.16:0/3203852438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a1e0a0 msgr2=0x5562a0ded440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a1e0a0 0x5562a0ded440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 -- 172.21.15.16:0/3203852438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 msgr2=0x5562a0ded980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 0x5562a0ded980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 -- 172.21.15.16:0/3203852438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a0ab2540 con 0x5562a0a46f60 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0decf00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe274031040 tx=0x7fe274009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b0d7d1818ee1a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe26f7fe640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 0x7fe2680b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe2822c4640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 0x7fe2680b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe26cff9640 1 -- 172.21.15.16:0/3203852438 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274031b80 con 0x5562a0a46f60 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe26f7fe640 1 -- 172.21.15.16:0/3859355891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fe2700e72b0 con 0x7fe27c1083d0 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe26cff9640 1 -- 172.21.15.16:0/3203852438 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe274036030 con 0x5562a0a46f60 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.566+0000 7fe26cff9640 1 -- 172.21.15.16:0/3203852438 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2740045a0 con 0x5562a0a46f60 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3203852438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 msgr2=0x5562a0decf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0decf00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe274031040 tx=0x7fe274009a20 comp rx=0 tx=0).stop 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe2822c4640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 0x7fe2680b8b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5562a0debb80 tx=0x7fe27800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3203852438 shutdown_connections 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a26280 0x5562a0ded980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a1e0a0 0x5562a0ded440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3203852438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0decf00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3203852438 >> 172.21.15.16:0/3203852438 conn(0x5562a0a3b350 msgr2=0x5562a0deadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3203852438 shutdown_connections 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.567+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3203852438 wait complete. 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 -- start start 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 Processor -- start 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0e047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562a0bb55f0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5562a0cef010 con 0x5562a0a46f60 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5562a0dfb500 con 0x5562a0a26280 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe28aa9c740 10 client.0 _ll_register_callbacks cb 0x5562a0d4e340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe2822c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe2822c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46282/0 (socket says 172.21.15.16:46282) 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe2822c4640 1 -- 172.21.15.16:0/867516178 learned_addr learned my addr 172.21.15.16:0/867516178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.568+0000 7fe281ac3640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe282ac5640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0e047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 -- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 msgr2=0x5562a0df8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 -- 172.21.15.16:0/867516178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 msgr2=0x5562a0e047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0e047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 -- 172.21.15.16:0/867516178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a0d1ca00 con 0x5562a0a1e0a0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe282ac5640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0e047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe2822c4640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe278003d00 tx=0x7fe2780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce6178a1892209d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27802c060 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fe278017030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe274036030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe278033050 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe28aa9c740 10 client.15639 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.569+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562a0e074a0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe274036030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe278011440 con 0x5562a0a1e0a0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 10 client.15639 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 10 client.15639 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 20 client.15639 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a058e0 0x5562a0e0af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 4 client.15639 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562a0d66b60 con 0x5562a0a058e0 2024-09-21T02:07:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.570+0000 7fe281ac3640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a058e0 0x5562a0e0af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.571+0000 7fe281ac3640 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a058e0 0x5562a0e0af50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dadc5f7c8893c079 server_cookie=7318112a15bf3271 in_seq=0 out_seq=0 2024-09-21T02:07:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.571+0000 7fe265ffb640 10 client.15639 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.571+0000 7fe265ffb640 1 -- 172.21.15.16:0/867516178 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5562a0d66b60 con 0x5562a0a058e0 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.571+0000 7fe265ffb640 10 client.15639 handle_command_reply: tid=0 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.571+0000 7fe28aa9c740 1 client.15639 shutdown 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 msgr2=0x5562a0df89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe278003d00 tx=0x7fe2780046c0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 shutdown_connections 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a058e0 0x5562a0e0af50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe265ffb640 0 client.15639 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a46f60 0x5562a0e047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/867516178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0df89e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 >> 172.21.15.16:0/867516178 conn(0x5562a0a3b350 msgr2=0x5562a0df00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 shutdown_connections 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 1 -- 172.21.15.16:0/867516178 wait complete. 2024-09-21T02:07:24.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.572+0000 7fe28aa9c740 20 client.15639 trim_cache size 0 max 16384 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 Processor -- start 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 -- start start 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a058e0 0x5562a0e04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e09290 con 0x5562a0a1e0a0 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e09960 con 0x5562a0a058e0 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e0a030 con 0x5562a0a26280 2024-09-21T02:07:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.590+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46292/0 (socket says 172.21.15.16:46292) 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 -- 172.21.15.16:0/3979317198 learned_addr learned my addr 172.21.15.16:0/3979317198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe2822c4640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a058e0 0x5562a0e04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 -- 172.21.15.16:0/3979317198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 msgr2=0x5562a0df7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 -- 172.21.15.16:0/3979317198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a058e0 msgr2=0x5562a0e04760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a058e0 0x5562a0e04760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 -- 172.21.15.16:0/3979317198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a0defd60 con 0x5562a0a1e0a0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe282ac5640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe281ac3640 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe270004700 tx=0x7fe2700e97a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c73d176bdac394d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe267fff640 1 -- 172.21.15.16:0/3979317198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700cee60 con 0x5562a0a1e0a0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe267fff640 1 -- 172.21.15.16:0/3979317198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2700af030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.591+0000 7fe267fff640 1 -- 172.21.15.16:0/3979317198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700b1930 con 0x5562a0a1e0a0 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3979317198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 msgr2=0x5562a0e04ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe270004700 tx=0x7fe2700e97a0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3979317198 shutdown_connections 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a26280 0x5562a0df7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e04ca0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/3979317198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a058e0 0x5562a0e04760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3979317198 >> 172.21.15.16:0/3979317198 conn(0x5562a0a3b350 msgr2=0x5562a0df39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3979317198 shutdown_connections 2024-09-21T02:07:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- 172.21.15.16:0/3979317198 wait complete. 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.592+0000 7fe28aa9c740 1 -- start start 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 Processor -- start 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a058e0 0x5562a0e0d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a26280 0x5562a0e12c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e0add0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e0a700 con 0x5562a0a26280 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5562a0e0b4a0 con 0x5562a0a058e0 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe28aa9c740 10 client.0 _ll_register_callbacks cb 0x7fe278004300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46304/0 (socket says 172.21.15.16:46304) 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 -- 172.21.15.16:0/2527474438 learned_addr learned my addr 172.21.15.16:0/2527474438 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe282ac5640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a26280 0x5562a0e12c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe2822c4640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a058e0 0x5562a0e0d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 -- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a058e0 msgr2=0x5562a0e0d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a058e0 0x5562a0e0d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 -- 172.21.15.16:0/2527474438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a26280 msgr2=0x5562a0e12c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a26280 0x5562a0e12c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.593+0000 7fe281ac3640 1 -- 172.21.15.16:0/2527474438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a0e0e2b0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe281ac3640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe2700a8f00 tx=0x7fe2700ac960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f61ddf487079f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700cee60 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fe2700a90c0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2700af030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe2700eae00 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 10 client.15645 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562a0e768d0 con 0x5562a0a1e0a0 2024-09-21T02:07:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2700af030 con 0x5562a0a1e0a0 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe270098020 con 0x5562a0a1e0a0 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 10 client.15645 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 10 client.15645 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 20 client.15645 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a46f60 0x5562a0e79c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 4 client.15645 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.594+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 --> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] -- command(tid 0: {"prefix": "status"}) -- 0x5562a0d62610 con 0x5562a0a46f60 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.595+0000 7fe2822c4640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a46f60 0x5562a0e79c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.595+0000 7fe2822c4640 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a46f60 0x5562a0e79c50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a55c6f8c99cfdaec server_cookie=ff5585691201650b in_seq=0 out_seq=0 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.595+0000 7fe265ffb640 10 client.15645 ms_handle_connect on v2:172.21.15.161:6836/2373808610 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe265ffb640 1 -- 172.21.15.16:0/2527474438 <== mds.4 v2:172.21.15.161:6836/2373808610 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5562a0d62610 con 0x5562a0a46f60 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe265ffb640 10 client.15645 handle_command_reply: tid=0 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 client.15645 shutdown 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 msgr2=0x5562a0e0da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe2700a8f00 tx=0x7fe2700ac960 comp rx=0 tx=0).stop 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 shutdown_connections 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:6836/2373808610,v1:172.21.15.161:6837/2373808610] conn(0x5562a0a46f60 0x5562a0e79c50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5562a0a26280 0x5562a0e12c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5562a0a1e0a0 0x5562a0e0da30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 --2- 172.21.15.16:0/2527474438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5562a0a058e0 0x5562a0e0d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 >> 172.21.15.16:0/2527474438 conn(0x5562a0a3b350 msgr2=0x5562a0dea600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe265ffb640 0 client.15645 ms_handle_reset on v2:172.21.15.161:6836/2373808610 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 shutdown_connections 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 1 -- 172.21.15.16:0/2527474438 wait complete. 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.596+0000 7fe28aa9c740 20 client.15645 trim_cache size 0 max 16384 2024-09-21T02:07:24.601 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24655, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 124.763200083, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 194.018548181 2024-09-21T02:07:24.602 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 msgr2=0x7fe2680b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 0x7fe2680b8b30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5562a0debb80 tx=0x7fe27800b390 comp rx=0 tx=0).stop 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 msgr2=0x7fe27c19b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe27000b450 tx=0x7fe27000b910 comp rx=0 tx=0).stop 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 shutdown_connections 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fe2680b6620 0x7fe2680b8b30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe27c108d20 0x7fe27c19bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe27c1083d0 0x7fe27c19b700 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 --2- 172.21.15.16:0/3859355891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe27c102ae0 0x7fe27c19b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 >> 172.21.15.16:0/3859355891 conn(0x7fe27c079760 msgr2=0x7fe27c105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 shutdown_connections 2024-09-21T02:07:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:24.597+0000 7fe284528640 1 -- 172.21.15.16:0/3859355891 wait complete. 2024-09-21T02:07:25.458 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:25 smithi016 ceph-mon[31245]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 113 KiB/s wr, 51 op/s 2024-09-21T02:07:25.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.e status 2024-09-21T02:07:25.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:25 smithi123 ceph-mon[33453]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 113 KiB/s wr, 51 op/s 2024-09-21T02:07:25.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:25 smithi161 ceph-mon[34061]: pgmap v360: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 113 KiB/s wr, 51 op/s 2024-09-21T02:07:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 -- 172.21.15.16:0/1014288384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40103b60 msgr2=0x7f0b40103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/1014288384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40103b60 0x7f0b40103f70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009900 tx=0x7f0b3002a410 comp rx=0 tx=0).stop 2024-09-21T02:07:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 -- 172.21.15.16:0/1014288384 shutdown_connections 2024-09-21T02:07:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/1014288384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40109df0 0x7f0b40110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/1014288384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 0x7f0b401098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/1014288384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40103b60 0x7f0b40103f70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 -- 172.21.15.16:0/1014288384 >> 172.21.15.16:0/1014288384 conn(0x7f0b4007b850 msgr2=0x7f0b4007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.346+0000 7f0b45c5d640 1 -- 172.21.15.16:0/1014288384 shutdown_connections 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.347+0000 7f0b45c5d640 1 -- 172.21.15.16:0/1014288384 wait complete. 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.347+0000 7f0b45c5d640 1 Processor -- start 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.347+0000 7f0b45c5d640 1 -- start start 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40103b60 0x7f0b4019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 0x7f0b4019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401139c0 con 0x7f0b40109df0 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40113840 con 0x7f0b401094a0 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b45c5d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40113b40 con 0x7f0b40103b60 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46330/0 (socket says 172.21.15.16:46330) 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 -- 172.21.15.16:0/3034336362 learned_addr learned my addr 172.21.15.16:0/3034336362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40103b60 0x7f0b4019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3effd640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 0x7f0b4019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 -- 172.21.15.16:0/3034336362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40103b60 msgr2=0x7f0b4019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40103b60 0x7f0b4019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 -- 172.21.15.16:0/3034336362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 msgr2=0x7f0b4019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 0x7f0b4019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 -- 172.21.15.16:0/3034336362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401a2560 con 0x7f0b40109df0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.348+0000 7f0b3ffff640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800eaa0 tx=0x7f0b3800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.349+0000 7f0b3cff9640 1 -- 172.21.15.16:0/3034336362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38004d20 con 0x7f0b40109df0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.349+0000 7f0b3cff9640 1 -- 172.21.15.16:0/3034336362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b38005020 con 0x7f0b40109df0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.349+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b401a2850 con 0x7f0b40109df0 2024-09-21T02:07:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.349+0000 7f0b3cff9640 1 -- 172.21.15.16:0/3034336362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b380116d0 con 0x7f0b40109df0 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.349+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b401a8fb0 con 0x7f0b40109df0 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.351+0000 7f0b4c1d1740 1 Processor -- start 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.351+0000 7f0b3cff9640 1 -- 172.21.15.16:0/3034336362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f0b380051d0 con 0x7f0b40109df0 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.351+0000 7f0b4c1d1740 1 -- start start 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aee0f60 0x56436b286f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b287440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56436b1fec10 con 0x56436aec0280 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56436b1ee040 con 0x56436aee0f60 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56436b1f1e50 con 0x56436aeb80a0 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46332/0 (socket says 172.21.15.16:46332) 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 -- 172.21.15.16:0/1642091146 learned_addr learned my addr 172.21.15.16:0/1642091146 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aee0f60 0x56436b286f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b287440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 -- 172.21.15.16:0/1642091146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 msgr2=0x56436b287440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3cff9640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 0x7f0b280b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b287440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 -- 172.21.15.16:0/1642091146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aee0f60 msgr2=0x56436b286f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aee0f60 0x56436b286f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3effd640 1 -- 172.21.15.16:0/1642091146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56436af4c540 con 0x56436aec0280 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b287440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.352+0000 7f0b3ffff640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 0x7f0b280b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.353+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009330 tx=0x7f0b340094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70ba6cc9fbc49af server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.353+0000 7f0b3cff9640 1 -- 172.21.15.16:0/3034336362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0b380e7aa0 con 0x7f0b40109df0 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.353+0000 7f0b25ffb640 1 -- 172.21.15.16:0/1642091146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34017e60 con 0x56436aec0280 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.353+0000 7f0b25ffb640 1 -- 172.21.15.16:0/1642091146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b3401d070 con 0x56436aec0280 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.353+0000 7f0b25ffb640 1 -- 172.21.15.16:0/1642091146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34018830 con 0x56436aec0280 2024-09-21T02:07:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1642091146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 msgr2=0x56436b287980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b3ffff640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 0x7f0b280b8b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b380a9c10 tx=0x7f0b380e2040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009330 tx=0x7f0b340094d0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1642091146 shutdown_connections 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aec0280 0x56436b287980 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b287440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1642091146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aee0f60 0x56436b286f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1642091146 >> 172.21.15.16:0/1642091146 conn(0x56436aed5350 msgr2=0x56436b284df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1642091146 shutdown_connections 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1642091146 wait complete. 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 -- start start 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.354+0000 7f0b4c1d1740 1 Processor -- start 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b292ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aee0f60 0x56436b29e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56436b04f5f0 con 0x56436aeb80a0 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56436b189010 con 0x56436aec0280 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56436b295680 con 0x56436aee0f60 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b4c1d1740 10 client.0 _ll_register_callbacks cb 0x56436b1e8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46344/0 (socket says 172.21.15.16:46344) 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 -- 172.21.15.16:0/2534299959 learned_addr learned my addr 172.21.15.16:0/2534299959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b292ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 -- 172.21.15.16:0/2534299959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aee0f60 msgr2=0x56436b29e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aee0f60 0x56436b29e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 -- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 msgr2=0x56436b292ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b292ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.355+0000 7f0b3effd640 1 -- 172.21.15.16:0/2534299959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56436b1b6a00 con 0x56436aeb80a0 2024-09-21T02:07:26.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b3effd640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0b34004700 tx=0x7f0b34009180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d1010fa3f80197 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34026070 con 0x56436aeb80a0 2024-09-21T02:07:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0b340185d0 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b3401d070 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0b34017e60 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b4c1d1740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56436b2a1580 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b3401d070 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0b3402b7b0 con 0x56436aeb80a0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b4c1d1740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.356+0000 7f0b4c1d1740 10 client.15657 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b4c1d1740 20 client.15657 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae9f8e0 0x56436b2a2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b4c1d1740 4 client.15657 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56436b200b60 con 0x56436ae9f8e0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae9f8e0 0x56436b2a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae9f8e0 0x56436b2a2930 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc54f103e7e58587 server_cookie=b5fd6b654938c3b5 in_seq=0 out_seq=0 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.357+0000 7f0b1effd640 10 client.15657 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b1effd640 1 -- 172.21.15.16:0/2534299959 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56436b200b60 con 0x56436ae9f8e0 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b1effd640 10 client.15657 handle_command_reply: tid=0 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 client.15657 shutdown 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 msgr2=0x56436b292980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0b34004700 tx=0x7f0b34009180 comp rx=0 tx=0).stop 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 shutdown_connections 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae9f8e0 0x56436b2a2930 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aee0f60 0x56436b29e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b292ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/2534299959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aeb80a0 0x56436b292980 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 >> 172.21.15.16:0/2534299959 conn(0x56436aed5350 msgr2=0x56436b28a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b1effd640 0 client.15657 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 shutdown_connections 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.358+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/2534299959 wait complete. 2024-09-21T02:07:26.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.359+0000 7f0b4c1d1740 20 client.15657 trim_cache size 0 max 16384 2024-09-21T02:07:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.376+0000 7f0b4c1d1740 1 Processor -- start 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.376+0000 7f0b4c1d1740 1 -- start start 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b28d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b29e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a2e80 con 0x56436aee0f60 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a3550 con 0x56436aec0280 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a3c20 con 0x56436aeb80a0 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46350/0 (socket says 172.21.15.16:46350) 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 -- 172.21.15.16:0/1856863071 learned_addr learned my addr 172.21.15.16:0/1856863071 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b28d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b29e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 -- 172.21.15.16:0/1856863071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 msgr2=0x56436b28d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b28d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 -- 172.21.15.16:0/1856863071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 msgr2=0x56436b29e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b29e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 -- 172.21.15.16:0/1856863071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56436b2891f0 con 0x56436aee0f60 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b29e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.377+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0b34015e00 tx=0x7f0b34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413ea5726ac67279 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b257fa640 1 -- 172.21.15.16:0/1856863071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34026040 con 0x56436aee0f60 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b257fa640 1 -- 172.21.15.16:0/1856863071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b3401d040 con 0x56436aee0f60 2024-09-21T02:07:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b257fa640 1 -- 172.21.15.16:0/1856863071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34018b50 con 0x56436aee0f60 2024-09-21T02:07:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1856863071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 msgr2=0x56436b29ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0b34015e00 tx=0x7f0b34008040 comp rx=0 tx=0).stop 2024-09-21T02:07:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1856863071 shutdown_connections 2024-09-21T02:07:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b29ee50 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b29e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1856863071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b28d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1856863071 >> 172.21.15.16:0/1856863071 conn(0x56436aed5350 msgr2=0x56436b292960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1856863071 shutdown_connections 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.378+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1856863071 wait complete. 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 -- start start 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 Processor -- start 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b2a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b2a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a49c0 con 0x56436aee0f60 2024-09-21T02:07:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a42f0 con 0x56436aec0280 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.379+0000 7f0b4c1d1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56436b2a5090 con 0x56436aeb80a0 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b4c1d1740 10 client.0 _ll_register_callbacks cb 0x7f0b34009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46360/0 (socket says 172.21.15.16:46360) 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 -- 172.21.15.16:0/1649692426 learned_addr learned my addr 172.21.15.16:0/1649692426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b2a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b2a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 -- 172.21.15.16:0/1649692426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 msgr2=0x56436b2a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b2a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 -- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 msgr2=0x56436b2a6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b2a6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 -- 172.21.15.16:0/1649692426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56436b2ac080 con 0x56436aee0f60 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b2a6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b3effd640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0b340092c0 tx=0x7f0b34009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346c9cc13b9d0d82 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b1effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34026040 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0b340188d0 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.380+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b3401d040 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0b34018ad0 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 10 client.15663 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56436b2ad3b0 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0b3401d040 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0b34036820 con 0x56436aee0f60 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 10 client.15663 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 10 client.15663 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 20 client.15663 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:26.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae74c90 0x56436b312ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 4 client.15663 mds_command: new command op to 24659 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 --> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] -- command(tid 0: {"prefix": "status"}) -- 0x56436b1fc610 con 0x56436ae74c90 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.381+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae74c90 0x56436b312ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b1effd640 10 client.15663 ms_handle_connect on v2:172.21.15.123:6834/2421556237 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b3f7fe640 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae74c90 0x56436b312ae0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=90f7a44ca0ab03db server_cookie=c89fc709c10cc386 in_seq=0 out_seq=0 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b1effd640 1 -- 172.21.15.16:0/1649692426 <== mds.3 v2:172.21.15.123:6834/2421556237 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56436b1fc610 con 0x56436ae74c90 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b1effd640 10 client.15663 handle_command_reply: tid=0 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 client.15663 shutdown 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 msgr2=0x56436b2aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0b340092c0 tx=0x7f0b34009110 comp rx=0 tx=0).stop 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 shutdown_connections 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:6834/2421556237,v1:172.21.15.123:6835/2421556237] conn(0x56436ae74c90 0x56436b312ae0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56436aee0f60 0x56436b2aba10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56436aec0280 0x56436b2a6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 --2- 172.21.15.16:0/1649692426 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56436aeb80a0 0x56436b2a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 >> 172.21.15.16:0/1649692426 conn(0x56436aed5350 msgr2=0x56436b28d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b1effd640 0 client.15663 ms_handle_reset on v2:172.21.15.123:6834/2421556237 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 shutdown_connections 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.382+0000 7f0b4c1d1740 1 -- 172.21.15.16:0/1649692426 wait complete. 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b4c1d1740 20 client.15663 trim_cache size 0 max 16384 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-09-21T02:07:26.390 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24659, 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 127.556881985, 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 201.048750523 2024-09-21T02:07:26.391 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 msgr2=0x7f0b280b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 0x7f0b280b8b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b380a9c10 tx=0x7f0b380e2040 comp rx=0 tx=0).stop 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 msgr2=0x7f0b401a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800eaa0 tx=0x7f0b3800eea0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 shutdown_connections 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0b280b65f0 0x7f0b280b8b00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b40109df0 0x7f0b401a1e30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b401094a0 0x7f0b4019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 --2- 172.21.15.16:0/3034336362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b40103b60 0x7f0b4019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 >> 172.21.15.16:0/3034336362 conn(0x7f0b4007b850 msgr2=0x7f0b40107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 shutdown_connections 2024-09-21T02:07:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:26.383+0000 7f0b45c5d640 1 -- 172.21.15.16:0/3034336362 wait complete. 2024-09-21T02:07:27.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.i status 2024-09-21T02:07:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:27 smithi123 ceph-mon[33453]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T02:07:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:27 smithi016 ceph-mon[31245]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T02:07:27.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:27 smithi161 ceph-mon[34061]: pgmap v361: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 -- 172.21.15.16:0/1590350169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 msgr2=0x7f0aa41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 --2- 172.21.15.16:0/1590350169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa41098b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009900 tx=0x7f0a9402a410 comp rx=0 tx=0).stop 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 -- 172.21.15.16:0/1590350169 shutdown_connections 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 --2- 172.21.15.16:0/1590350169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4109df0 0x7f0aa4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 --2- 172.21.15.16:0/1590350169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa41098b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 --2- 172.21.15.16:0/1590350169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4103b60 0x7f0aa4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 -- 172.21.15.16:0/1590350169 >> 172.21.15.16:0/1590350169 conn(0x7f0aa407b850 msgr2=0x7f0aa407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 -- 172.21.15.16:0/1590350169 shutdown_connections 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.076+0000 7f0aab384640 1 -- 172.21.15.16:0/1590350169 wait complete. 2024-09-21T02:07:28.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 Processor -- start 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 -- start start 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4103b60 0x7f0aa419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4109df0 0x7f0aa419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa4113960 con 0x7f0aa41094a0 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa41137e0 con 0x7f0aa4109df0 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aab384640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa4113ae0 con 0x7f0aa4103b60 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49504/0 (socket says 172.21.15.16:49504) 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 -- 172.21.15.16:0/2206112867 learned_addr learned my addr 172.21.15.16:0/2206112867 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa9120640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4103b60 0x7f0aa419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa9921640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4109df0 0x7f0aa419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 -- 172.21.15.16:0/2206112867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4103b60 msgr2=0x7f0aa419a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4103b60 0x7f0aa419a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 -- 172.21.15.16:0/2206112867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4109df0 msgr2=0x7f0aa419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4109df0 0x7f0aa419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.077+0000 7f0aa891f640 1 -- 172.21.15.16:0/2206112867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa41a0340 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0a94004770 tx=0x7f0a94004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0a9a7fc640 1 -- 172.21.15.16:0/2206112867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9403f030 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa41a05d0 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0a9a7fc640 1 -- 172.21.15.16:0/2206112867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a94004440 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0a9a7fc640 1 -- 172.21.15.16:0/2206112867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94035070 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.078+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa41a6d90 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.080+0000 7f0ab18f8740 1 Processor -- start 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 -- start start 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0a9a7fc640 1 -- 172.21.15.16:0/2206112867 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f0a94042030 con 0x7f0aa41094a0 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa36f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 0x55c8caa37440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8ca9aec10 con 0x55c8ca670280 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c8ca99e040 con 0x55c8ca690f60 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c8ca9a1e50 con 0x55c8ca6680a0 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49520/0 (socket says 172.21.15.16:49520) 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 -- 172.21.15.16:0/2405237532 learned_addr learned my addr 172.21.15.16:0/2405237532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa9921640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa36f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa9120640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 0x55c8caa37440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 -- 172.21.15.16:0/2405237532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 msgr2=0x55c8caa37440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 0x55c8caa37440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 -- 172.21.15.16:0/2405237532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 msgr2=0x55c8caa36f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa36f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa891f640 1 -- 172.21.15.16:0/2405237532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8ca6fc540 con 0x55c8ca670280 2024-09-21T02:07:28.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.081+0000 7f0aa9120640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 0x55c8caa37440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0aa891f640 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0a94105310 tx=0x7f0a94110220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a501c5e79acdb0c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0a9a7fc640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 0x7f0a900b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0aa9921640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 0x7f0a900b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0a8f7fe640 1 -- 172.21.15.16:0/2405237532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940ca0a0 con 0x55c8ca670280 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0a9a7fc640 1 -- 172.21.15.16:0/2206112867 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0a94107eb0 con 0x7f0aa41094a0 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0a8f7fe640 1 -- 172.21.15.16:0/2405237532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a94115070 con 0x55c8ca670280 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.082+0000 7f0a8f7fe640 1 -- 172.21.15.16:0/2405237532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940cd1c0 con 0x55c8ca670280 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- 172.21.15.16:0/2405237532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 msgr2=0x55c8caa37980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0a94105310 tx=0x7f0a94110220 comp rx=0 tx=0).stop 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0aa9921640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 0x7f0a900b8b00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c005fd0 tx=0x7f0a9c009740 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- 172.21.15.16:0/2405237532 shutdown_connections 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa37980 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca6680a0 0x55c8caa37440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/2405237532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa36f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- 172.21.15.16:0/2405237532 >> 172.21.15.16:0/2405237532 conn(0x55c8ca685350 msgr2=0x55c8caa34df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- 172.21.15.16:0/2405237532 shutdown_connections 2024-09-21T02:07:28.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- 172.21.15.16:0/2405237532 wait complete. 2024-09-21T02:07:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.083+0000 7f0ab18f8740 1 -- start start 2024-09-21T02:07:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 Processor -- start 2024-09-21T02:07:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca670280 0x55c8caa42f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa4e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8ca7ff5f0 con 0x55c8ca6680a0 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c8ca939010 con 0x55c8ca690f60 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa454a0 con 0x55c8ca670280 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0ab18f8740 10 client.0 _ll_register_callbacks cb 0x55c8ca998340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0aa9120640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0aa9120640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49532/0 (socket says 172.21.15.16:49532) 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0aa9120640 1 -- 172.21.15.16:0/3745151425 learned_addr learned my addr 172.21.15.16:0/3745151425 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0aa891f640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca670280 0x55c8caa42f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.084+0000 7f0aa9921640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa4e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 -- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca670280 msgr2=0x55c8caa42f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca670280 0x55c8caa42f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 -- 172.21.15.16:0/3745151425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 msgr2=0x55c8caa4e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa4e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 -- 172.21.15.16:0/3745151425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8ca966a00 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0aa9120640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0013790 tx=0x7f0aa0013b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9f3c40d7df088d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0a8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0011d50 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0aa0004020 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a94115070 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0aa0026050 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0ab18f8740 10 client.15675 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.085+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8caa513b0 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a94115070 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0aa001a030 con 0x55c8ca6680a0 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 10 client.15675 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 10 client.15675 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:07:28.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 20 client.15675 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca64f8e0 0x55c8caa54ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 4 client.15675 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8ca9b0b60 con 0x55c8ca64f8e0 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.086+0000 7f0aa891f640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca64f8e0 0x55c8caa54ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.087+0000 7f0a8cff9640 10 client.15675 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.087+0000 7f0aa891f640 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca64f8e0 0x55c8caa54ea0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d73b23c7fa6b1d13 server_cookie=c99b518841a18899 in_seq=0 out_seq=0 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0a8cff9640 1 -- 172.21.15.16:0/3745151425 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c8ca9b0b60 con 0x55c8ca64f8e0 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0a8cff9640 10 client.15675 handle_command_reply: tid=0 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 client.15675 shutdown 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 msgr2=0x55c8caa429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0013790 tx=0x7f0aa0013b90 comp rx=0 tx=0).stop 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 shutdown_connections 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca64f8e0 0x55c8caa54ea0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca690f60 0x55c8caa4e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca670280 0x55c8caa42f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/3745151425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa429f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 >> 172.21.15.16:0/3745151425 conn(0x55c8ca685350 msgr2=0x55c8caa3a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0a8cff9640 0 client.15675 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 shutdown_connections 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 1 -- 172.21.15.16:0/3745151425 wait complete. 2024-09-21T02:07:28.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.088+0000 7f0ab18f8740 20 client.15675 trim_cache size 0 max 16384 2024-09-21T02:07:28.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.106+0000 7f0ab18f8740 1 Processor -- start 2024-09-21T02:07:28.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.106+0000 7f0ab18f8740 1 -- start start 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 0x55c8caa3d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa531e0 con 0x55c8ca670280 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa538b0 con 0x55c8ca6680a0 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa53f80 con 0x55c8ca690f60 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49548/0 (socket says 172.21.15.16:49548) 2024-09-21T02:07:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 -- 172.21.15.16:0/4009514255 learned_addr learned my addr 172.21.15.16:0/4009514255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa9921640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 -- 172.21.15.16:0/4009514255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 msgr2=0x55c8caa415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa9120640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 0x55c8caa3d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 -- 172.21.15.16:0/4009514255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 msgr2=0x55c8caa3d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 0x55c8caa3d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 -- 172.21.15.16:0/4009514255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8caa2f080 con 0x55c8ca670280 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa9120640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 0x55c8caa3d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.107+0000 7f0aa891f640 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000c00 tx=0x7f0a94003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749f00a0934a0fc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0a8effd640 1 -- 172.21.15.16:0/4009514255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94049040 con 0x55c8ca670280 2024-09-21T02:07:28.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0a8effd640 1 -- 172.21.15.16:0/4009514255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a94115040 con 0x55c8ca670280 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0a8effd640 1 -- 172.21.15.16:0/4009514255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940cd3a0 con 0x55c8ca670280 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 -- 172.21.15.16:0/4009514255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 msgr2=0x55c8caa41070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000c00 tx=0x7f0a94003bf0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 -- 172.21.15.16:0/4009514255 shutdown_connections 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca670280 0x55c8caa41070 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.108+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/4009514255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca6680a0 0x55c8caa3d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.109+0000 7f0ab18f8740 1 -- 172.21.15.16:0/4009514255 >> 172.21.15.16:0/4009514255 conn(0x55c8ca685350 msgr2=0x55c8caa429d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.109+0000 7f0ab18f8740 1 -- 172.21.15.16:0/4009514255 shutdown_connections 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.109+0000 7f0ab18f8740 1 -- 172.21.15.16:0/4009514255 wait complete. 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.109+0000 7f0ab18f8740 1 -- start start 2024-09-21T02:07:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.109+0000 7f0ab18f8740 1 Processor -- start 2024-09-21T02:07:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca670280 0x55c8caa56c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa54d20 con 0x55c8ca6680a0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa54650 con 0x55c8ca670280 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c8caa553f0 con 0x55c8ca690f60 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49554/0 (socket says 172.21.15.16:49554) 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 -- 172.21.15.16:0/345259069 learned_addr learned my addr 172.21.15.16:0/345259069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0ab18f8740 10 client.0 _ll_register_callbacks cb 0x7f0aa0009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa891f640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca670280 0x55c8caa56c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9921640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 -- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 msgr2=0x55c8caa571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 -- 172.21.15.16:0/345259069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca670280 msgr2=0x55c8caa56c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca670280 0x55c8caa56c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 -- 172.21.15.16:0/345259069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8caa577c0 con 0x55c8ca6680a0 2024-09-21T02:07:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0aa9120640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009830 tx=0x7f0aa0009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e6b9ce11c6d815 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.110+0000 7f0a8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000f6a0 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f0aa0011d60 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a94115040 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0aa0039050 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 10 client.15681 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8caa5d110 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a94115040 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0aa001a040 con 0x55c8ca6680a0 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 10 client.15681 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 10 client.15681 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 20 client.15681 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca624c90 0x55c8caac39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 4 client.15681 mds_command: new command op to 24667 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.111+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 --> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] -- command(tid 0: {"prefix": "status"}) -- 0x55c8ca9ac610 con 0x55c8ca624c90 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0aa891f640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca624c90 0x55c8caac39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0a8cff9640 10 client.15681 ms_handle_connect on v2:172.21.15.161:6838/2696881795 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0aa891f640 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca624c90 0x55c8caac39b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=341e544395e39ec5 server_cookie=4393a5fa969e75d3 in_seq=0 out_seq=0 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0a8cff9640 1 -- 172.21.15.16:0/345259069 <== mds.2 v2:172.21.15.161:6838/2696881795 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c8ca9ac610 con 0x55c8ca624c90 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0a8cff9640 10 client.15681 handle_command_reply: tid=0 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0ab18f8740 1 client.15681 shutdown 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 msgr2=0x55c8caa5d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.112+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009830 tx=0x7f0aa0009e10 comp rx=0 tx=0).stop 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 shutdown_connections 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c8ca690f60 0x55c8caa571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.161:6838/2696881795,v1:172.21.15.161:6839/2696881795] conn(0x55c8ca624c90 0x55c8caac39b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c8ca670280 0x55c8caa56c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 --2- 172.21.15.16:0/345259069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c8ca6680a0 0x55c8caa5d790 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0a8cff9640 0 client.15681 ms_handle_reset on v2:172.21.15.161:6838/2696881795 2024-09-21T02:07:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 >> 172.21.15.16:0/345259069 conn(0x55c8ca685350 msgr2=0x55c8caa34600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 shutdown_connections 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 1 -- 172.21.15.16:0/345259069 wait complete. 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.113+0000 7f0ab18f8740 20 client.15681 trim_cache size 0 max 16384 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24667, 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:28.118 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 130.29799056600001, 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 192.32296457300001 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 msgr2=0x7f0a900b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 0x7f0a900b8b00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c005fd0 tx=0x7f0a9c009740 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 msgr2=0x7f0aa419a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0a94004770 tx=0x7f0a94004040 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 shutdown_connections 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b65f0 0x7f0a900b8b00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa4109df0 0x7f0aa419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa41094a0 0x7f0aa419a970 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 --2- 172.21.15.16:0/2206112867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa4103b60 0x7f0aa419a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 >> 172.21.15.16:0/2206112867 conn(0x7f0aa407b850 msgr2=0x7f0aa41075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 shutdown_connections 2024-09-21T02:07:28.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:28.114+0000 7f0aab384640 1 -- 172.21.15.16:0/2206112867 wait complete. 2024-09-21T02:07:28.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.h status 2024-09-21T02:07:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:29.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:29 smithi123 ceph-mon[33453]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:29.586 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:29 smithi016 ceph-mon[31245]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:29.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:29 smithi161 ceph-mon[34061]: pgmap v362: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 -- 172.21.15.16:0/2931225492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb01031d0 msgr2=0x7f6cb01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/2931225492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb01031d0 0x7f6cb01035e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009900 tx=0x7f6ca002a410 comp rx=0 tx=0).stop 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 -- 172.21.15.16:0/2931225492 shutdown_connections 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/2931225492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb0108f60 0x7f6cb010f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/2931225492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108610 0x7f6cb0108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/2931225492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb01031d0 0x7f6cb01035e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 -- 172.21.15.16:0/2931225492 >> 172.21.15.16:0/2931225492 conn(0x7f6cb0079e40 msgr2=0x7f6cb007a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 -- 172.21.15.16:0/2931225492 shutdown_connections 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.681+0000 7f6cb4e49640 1 -- 172.21.15.16:0/2931225492 wait complete. 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 Processor -- start 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 -- start start 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb01031d0 0x7f6cb019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 0x7f6cb01a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb0113bf0 con 0x7f6cb0108610 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb0113a70 con 0x7f6cb01031d0 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cb4e49640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb0113d70 con 0x7f6cb0108f60 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49574/0 (socket says 172.21.15.16:49574) 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cadd9b640 1 -- 172.21.15.16:0/42343096 learned_addr learned my addr 172.21.15.16:0/42343096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6cae59c640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb01031d0 0x7f6cb019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.682+0000 7f6caed9d640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 0x7f6cb01a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 -- 172.21.15.16:0/42343096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 msgr2=0x7f6cb01a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 0x7f6cb01a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 -- 172.21.15.16:0/42343096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb01031d0 msgr2=0x7f6cb019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb01031d0 0x7f6cb019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 -- 172.21.15.16:0/42343096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb01a27f0 con 0x7f6cb0108610 2024-09-21T02:07:29.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6caed9d640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 0x7f6cb01a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400e9c0 tx=0x7f6ca400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6c9f7fe640 1 -- 172.21.15.16:0/42343096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400ccc0 con 0x7f6cb0108610 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6c9f7fe640 1 -- 172.21.15.16:0/42343096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ca40054b0 con 0x7f6cb0108610 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb01a2ae0 con 0x7f6cb0108610 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6c9f7fe640 1 -- 172.21.15.16:0/42343096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4010430 con 0x7f6cb0108610 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.683+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb01a9240 con 0x7f6cb0108610 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 Processor -- start 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 -- start start 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54d3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54d4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585f544bc10 con 0x5585f51050a0 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5585f543b040 con 0x5585f510d280 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5585f543ee50 con 0x5585f512df60 2024-09-21T02:07:29.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.686+0000 7f6c9f7fe640 1 -- 172.21.15.16:0/42343096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f6ca4004f00 con 0x7f6cb0108610 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49588/0 (socket says 172.21.15.16:49588) 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 -- 172.21.15.16:0/3564219269 learned_addr learned my addr 172.21.15.16:0/3564219269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6caed9d640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54d3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54d4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 -- 172.21.15.16:0/3564219269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 msgr2=0x5585f54d3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54d3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 -- 172.21.15.16:0/3564219269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 msgr2=0x5585f54d4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54d4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 -- 172.21.15.16:0/3564219269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f5199540 con 0x5585f51050a0 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54d4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6cae59c640 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ca002a930 tx=0x7f6ca0031a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a3e14655a840c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6c9cff9640 1 -- 172.21.15.16:0/3564219269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0035030 con 0x5585f51050a0 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.687+0000 7f6c9cff9640 1 -- 172.21.15.16:0/3564219269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ca0040070 con 0x5585f51050a0 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6c9cff9640 1 -- 172.21.15.16:0/3564219269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0003850 con 0x5585f51050a0 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6c9f7fe640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 0x7f6c980b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6caed9d640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 0x7f6c980b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6c9f7fe640 1 -- 172.21.15.16:0/42343096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f6ca40e7d10 con 0x7f6cb0108610 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/3564219269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 msgr2=0x5585f54d4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ca002a930 tx=0x7f6ca0031a70 comp rx=0 tx=0).stop 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6caed9d640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 0x7f6c980b8b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005fd0 tx=0x7f6ca8009500 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/3564219269 shutdown_connections 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54d4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f51050a0 0x5585f54d4440 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/3564219269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54d3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.688+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/3564219269 >> 172.21.15.16:0/3564219269 conn(0x5585f5122350 msgr2=0x5585f54d1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/3564219269 shutdown_connections 2024-09-21T02:07:29.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/3564219269 wait complete. 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 -- start start 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 Processor -- start 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f51050a0 0x5585f54df9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.689+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54dff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585f529c5f0 con 0x5585f512df60 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5585f53d6010 con 0x5585f510d280 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5585f54e2680 con 0x5585f51050a0 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 10 client.0 _ll_register_callbacks cb 0x5585f5435340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49596/0 (socket says 172.21.15.16:49596) 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 -- 172.21.15.16:0/319285428 learned_addr learned my addr 172.21.15.16:0/319285428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:29.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cae59c640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f51050a0 0x5585f54df9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 -- 172.21.15.16:0/319285428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f51050a0 msgr2=0x5585f54df9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f51050a0 0x5585f54df9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 -- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 msgr2=0x5585f54dff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54dff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 -- 172.21.15.16:0/319285428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f5403a00 con 0x5585f512df60 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6caed9d640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005cc0 tx=0x7f6ca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea3e2719840e092 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca801fc80 con 0x5585f512df60 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f6ca8017030 con 0x5585f512df60 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.690+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ca0040070 con 0x5585f512df60 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ca8032050 con 0x5585f512df60 2024-09-21T02:07:29.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 10 client.15693 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585f54ee570 con 0x5585f512df60 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ca0040070 con 0x5585f512df60 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f6ca801c3f0 con 0x5585f512df60 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 10 client.15693 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 10 client.15693 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 20 client.15693 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50ec8e0 0x5585f54f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 4 client.15693 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.691+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585f544db60 con 0x5585f50ec8e0 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.692+0000 7f6cae59c640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50ec8e0 0x5585f54f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.692+0000 7f6cae59c640 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50ec8e0 0x5585f54f20c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83fc62f44d99c975 server_cookie=a44791581c14ce6 in_seq=0 out_seq=0 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.692+0000 7f6c95ffb640 10 client.15693 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6c95ffb640 1 -- 172.21.15.16:0/319285428 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5585f544db60 con 0x5585f50ec8e0 2024-09-21T02:07:29.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6c95ffb640 10 client.15693 handle_command_reply: tid=0 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 client.15693 shutdown 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 msgr2=0x5585f54eb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005cc0 tx=0x7f6ca8004a40 comp rx=0 tx=0).stop 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 shutdown_connections 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50ec8e0 0x5585f54f20c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54eb9a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f510d280 0x5585f54dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6c95ffb640 0 client.15693 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/319285428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f51050a0 0x5585f54df9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 >> 172.21.15.16:0/319285428 conn(0x5585f5122350 msgr2=0x5585f54d7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 shutdown_connections 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/319285428 wait complete. 2024-09-21T02:07:29.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.693+0000 7f6cbb3bd740 20 client.15693 trim_cache size 0 max 16384 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 Processor -- start 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 -- start start 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54da020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f510d280 0x5585f54eb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f0400 con 0x5585f512df60 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f0ad0 con 0x5585f51050a0 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.712+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f11a0 con 0x5585f510d280 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49608/0 (socket says 172.21.15.16:49608) 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 -- 172.21.15.16:0/2400123453 learned_addr learned my addr 172.21.15.16:0/2400123453 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f510d280 0x5585f54eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6cae59c640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 -- 172.21.15.16:0/2400123453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f510d280 msgr2=0x5585f54eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f510d280 0x5585f54eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 -- 172.21.15.16:0/2400123453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 msgr2=0x5585f54da020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54da020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 -- 172.21.15.16:0/2400123453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f54d6790 con 0x5585f512df60 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6cae59c640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54da020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6caed9d640 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005d70 tx=0x7f6ca801f800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd45118ecdf6b3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6c97fff640 1 -- 172.21.15.16:0/2400123453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca801c5f0 con 0x5585f512df60 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6c97fff640 1 -- 172.21.15.16:0/2400123453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ca801ca30 con 0x5585f512df60 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.713+0000 7f6c97fff640 1 -- 172.21.15.16:0/2400123453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80055e0 con 0x5585f512df60 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/2400123453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 msgr2=0x5585f54ebe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005d70 tx=0x7f6ca801f800 comp rx=0 tx=0).stop 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/2400123453 shutdown_connections 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f512df60 0x5585f54ebe50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f510d280 0x5585f54eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/2400123453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54da020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/2400123453 >> 172.21.15.16:0/2400123453 conn(0x5585f5122350 msgr2=0x5585f54da900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/2400123453 shutdown_connections 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.714+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/2400123453 wait complete. 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 -- start start 2024-09-21T02:07:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 Processor -- start 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54f5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f1f40 con 0x5585f510d280 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f1870 con 0x5585f51050a0 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5585f54f2610 con 0x5585f512df60 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.715+0000 7f6cbb3bd740 10 client.0 _ll_register_callbacks cb 0x7f6ca8009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49622/0 (socket says 172.21.15.16:49622) 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 -- 172.21.15.16:0/978967936 learned_addr learned my addr 172.21.15.16:0/978967936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cae59c640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6caed9d640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 -- 172.21.15.16:0/978967936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 msgr2=0x5585f54f5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54f5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 -- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 msgr2=0x5585f54f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 -- 172.21.15.16:0/978967936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f54f9f10 con 0x5585f510d280 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6caed9d640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cae59c640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cadd9b640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004650 tx=0x7f6ca40e56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59b64c1037db0db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca40ad020 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f6ca40aebe0 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ca801ca30 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.716+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ca40ad3c0 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 10 client.15699 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585f555d9a0 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ca801ca30 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f6ca40a9200 con 0x5585f510d280 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 10 client.15699 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 10 client.15699 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 20 client.15699 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50c1c90 0x5585f55612a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 4 client.15699 mds_command: new command op to 24683 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 --> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] -- command(tid 0: {"prefix": "status"}) -- 0x5585f5449610 con 0x5585f50c1c90 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.717+0000 7f6cae59c640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50c1c90 0x5585f55612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cae59c640 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50c1c90 0x5585f55612a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=269a069879e69c7a server_cookie=be1823854d2dbac in_seq=0 out_seq=0 2024-09-21T02:07:29.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6c95ffb640 10 client.15699 ms_handle_connect on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:29.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6c95ffb640 1 -- 172.21.15.16:0/978967936 <== mds.1 v2:172.21.15.123:6836/1023560595 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5585f5449610 con 0x5585f50c1c90 2024-09-21T02:07:29.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6c95ffb640 10 client.15699 handle_command_reply: tid=0 2024-09-21T02:07:29.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 client.15699 shutdown 2024-09-21T02:07:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 msgr2=0x5585f54f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004650 tx=0x7f6ca40e56b0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 shutdown_connections 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:6836/1023560595,v1:172.21.15.123:6837/1023560595] conn(0x5585f50c1c90 0x5585f55612a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5585f512df60 0x5585f54f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5585f510d280 0x5585f54f4ae0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6c95ffb640 0 client.15699 ms_handle_reset on v2:172.21.15.123:6836/1023560595 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 --2- 172.21.15.16:0/978967936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5585f51050a0 0x5585f54f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 >> 172.21.15.16:0/978967936 conn(0x5585f5122350 msgr2=0x5585f54d4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 shutdown_connections 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 1 -- 172.21.15.16:0/978967936 wait complete. 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.718+0000 7f6cbb3bd740 20 client.15699 trim_cache size 0 max 16384 2024-09-21T02:07:29.726 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24683, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 132.91185861, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 196.511663852 2024-09-21T02:07:29.727 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 msgr2=0x7f6c980b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 0x7f6c980b8b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8005fd0 tx=0x7f6ca8009500 comp rx=0 tx=0).stop 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 msgr2=0x7f6cb019ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400e9c0 tx=0x7f6ca400ee80 comp rx=0 tx=0).stop 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 shutdown_connections 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f6c980b6670 0x7f6c980b8b80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb0108f60 0x7f6cb01a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb0108610 0x7f6cb019ce20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 --2- 172.21.15.16:0/42343096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb01031d0 0x7f6cb019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 >> 172.21.15.16:0/42343096 conn(0x7f6cb0079e40 msgr2=0x7f6cb0106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 shutdown_connections 2024-09-21T02:07:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:29.719+0000 7f6cb4e49640 1 -- 172.21.15.16:0/42343096 wait complete. 2024-09-21T02:07:30.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph tell mds.b status 2024-09-21T02:07:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 -- 172.21.15.16:0/1034856317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514108cf0 msgr2=0x7fd51410f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 --2- 172.21.15.16:0/1034856317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514108cf0 0x7fd51410f520 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009ca0 tx=0x7fd50c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 -- 172.21.15.16:0/1034856317 shutdown_connections 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 --2- 172.21.15.16:0/1034856317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514108cf0 0x7fd51410f520 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 --2- 172.21.15.16:0/1034856317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5141083a0 0x7fd5141087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 --2- 172.21.15.16:0/1034856317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd514102a60 0x7fd514102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.205+0000 7fd51a912640 1 -- 172.21.15.16:0/1034856317 >> 172.21.15.16:0/1034856317 conn(0x7fd514079730 msgr2=0x7fd514079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 -- 172.21.15.16:0/1034856317 shutdown_connections 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 -- 172.21.15.16:0/1034856317 wait complete. 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 Processor -- start 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 -- start start 2024-09-21T02:07:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.206+0000 7fd51a912640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 0x7fd5140ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514108cf0 0x7fd5141000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd514112860 con 0x7fd514102a60 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd5141126e0 con 0x7fd5141083a0 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5141129e0 con 0x7fd514108cf0 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49646/0 (socket says 172.21.15.16:49646) 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 -- 172.21.15.16:0/689642475 learned_addr learned my addr 172.21.15.16:0/689642475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd518eaf640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514108cf0 0x7fd5141000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd50bfff640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 0x7fd5140ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 -- 172.21.15.16:0/689642475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514108cf0 msgr2=0x7fd5141000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514108cf0 0x7fd5141000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 -- 172.21.15.16:0/689642475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 msgr2=0x7fd5140ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 0x7fd5140ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 -- 172.21.15.16:0/689642475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5141a8680 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd50bfff640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 0x7fd5140ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd513fff640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd50400d950 tx=0x7fd50400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd511ffb640 1 -- 172.21.15.16:0/689642475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504017030 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd511ffb640 1 -- 172.21.15.16:0/689642475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd504004020 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd511ffb640 1 -- 172.21.15.16:0/689642475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5040055d0 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5141a8970 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.207+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5141a8e60 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 Processor -- start 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd511ffb640 1 -- 172.21.15.16:0/689642475 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fd504009a30 con 0x7fd514102a60 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 -- start start 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c83f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c84440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557301bfbc10 con 0x5573018bd280 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557301beb040 con 0x5573018ddf60 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.210+0000 7fd520e86740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557301beee50 con 0x5573018b50a0 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49656/0 (socket says 172.21.15.16:49656) 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 -- 172.21.15.16:0/3507123045 learned_addr learned my addr 172.21.15.16:0/3507123045 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd513fff640 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c84440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd518eaf640 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c83f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd511ffb640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 0x7fd5000b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 -- 172.21.15.16:0/3507123045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 msgr2=0x557301c84440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c84440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 -- 172.21.15.16:0/3507123045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 msgr2=0x557301c83f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c83f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 -- 172.21.15.16:0/3507123045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557301949540 con 0x5573018bd280 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd511ffb640 1 -- 172.21.15.16:0/689642475 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fd5040e73d0 con 0x7fd514102a60 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd518eaf640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 0x7fd5000b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009c20 tx=0x7fd4fc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fc7f0767fde2c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.211+0000 7fd50a7fc640 1 -- 172.21.15.16:0/3507123045 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc004030 con 0x5573018bd280 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd50a7fc640 1 -- 172.21.15.16:0/3507123045 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4fc0035f0 con 0x5573018bd280 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd50a7fc640 1 -- 172.21.15.16:0/3507123045 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc012690 con 0x5573018bd280 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 -- 172.21.15.16:0/3507123045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 msgr2=0x557301c84980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd518eaf640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 0x7fd5000b8b80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006010 tx=0x7fd50c0095c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009c20 tx=0x7fd4fc010740 comp rx=0 tx=0).stop 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 -- 172.21.15.16:0/3507123045 shutdown_connections 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c84980 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c84440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 --2- 172.21.15.16:0/3507123045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c83f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 -- 172.21.15.16:0/3507123045 >> 172.21.15.16:0/3507123045 conn(0x5573018d2350 msgr2=0x557301c81df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 -- 172.21.15.16:0/3507123045 shutdown_connections 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.212+0000 7fd520e86740 1 -- 172.21.15.16:0/3507123045 wait complete. 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 -- start start 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 Processor -- start 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c8f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c9ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557301a4c5f0 con 0x5573018bd280 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557301b86010 con 0x5573018ddf60 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.213+0000 7fd520e86740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557301c926e0 con 0x5573018b50a0 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd520e86740 10 client.0 _ll_register_callbacks cb 0x557301be5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49660/0 (socket says 172.21.15.16:49660) 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 -- 172.21.15.16:0/3948788204 learned_addr learned my addr 172.21.15.16:0/3948788204 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:31.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd513fff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c8f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 -- 172.21.15.16:0/3948788204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 msgr2=0x557301c8f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c8f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 -- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 msgr2=0x557301c9ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c9ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 -- 172.21.15.16:0/3948788204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557301bb3a00 con 0x5573018bd280 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc005780 tx=0x7fd4fc009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af90409890c376db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd4f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc02b060 con 0x5573018bd280 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fd4fc0039b0 con 0x5573018bd280 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4fc0035f0 con 0x5573018bd280 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4fc032050 con 0x5573018bd280 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd520e86740 10 client.15711 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:31.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.214+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557301c9e650 con 0x5573018bd280 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd4fc0035f0 con 0x5573018bd280 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd4fc00f040 con 0x5573018bd280 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 10 client.15711 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 10 client.15711 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 20 client.15711 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55730189c8e0 0x557301ca1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 4 client.15711 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557301bfdb60 con 0x55730189c8e0 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.215+0000 7fd513fff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55730189c8e0 0x557301ca1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.216+0000 7fd513fff640 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55730189c8e0 0x557301ca1d50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a79a006c40be1c53 server_cookie=717cbbbca22176a4 in_seq=0 out_seq=0 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.216+0000 7fd4f37fe640 10 client.15711 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.216+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3948788204 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557301bfdb60 con 0x55730189c8e0 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.216+0000 7fd4f37fe640 10 client.15711 handle_command_reply: tid=0 2024-09-21T02:07:31.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.216+0000 7fd520e86740 1 client.15711 shutdown 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 msgr2=0x557301c8ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc005780 tx=0x7fd4fc009d10 comp rx=0 tx=0).stop 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 shutdown_connections 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55730189c8e0 0x557301ca1d50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301c9ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301c8ff30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 --2- 172.21.15.16:0/3948788204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301c8f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd4f37fe640 0 client.15711 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 >> 172.21.15.16:0/3948788204 conn(0x5573018d2350 msgr2=0x557301c87090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 shutdown_connections 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 1 -- 172.21.15.16:0/3948788204 wait complete. 2024-09-21T02:07:31.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.217+0000 7fd520e86740 20 client.15711 trim_cache size 0 max 16384 2024-09-21T02:07:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.235+0000 7fd520e86740 1 Processor -- start 2024-09-21T02:07:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.235+0000 7fd520e86740 1 -- start start 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.235+0000 7fd520e86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd520e86740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018bd280 0x557301c8f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd520e86740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018ddf60 0x557301c8ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd520e86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557301ca0090 con 0x5573018b50a0 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd520e86740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557301ca0760 con 0x5573018bd280 2024-09-21T02:07:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd520e86740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557301ca0e30 con 0x5573018ddf60 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49670/0 (socket says 172.21.15.16:49670) 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 -- 172.21.15.16:0/2158080526 learned_addr learned my addr 172.21.15.16:0/2158080526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd50bfff640 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018bd280 0x557301c8f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd518eaf640 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018ddf60 0x557301c8ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 -- 172.21.15.16:0/2158080526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018ddf60 msgr2=0x557301c8ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018ddf60 0x557301c8ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 -- 172.21.15.16:0/2158080526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018bd280 msgr2=0x557301c8f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018bd280 0x557301c8f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 -- 172.21.15.16:0/2158080526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557301c86d60 con 0x5573018b50a0 2024-09-21T02:07:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.236+0000 7fd513fff640 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd5040aa1e0 tx=0x7fd5040eaf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0261431078760e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.239+0000 7fd509ffb640 1 -- 172.21.15.16:0/2158080526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5040acd40 con 0x5573018b50a0 2024-09-21T02:07:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.239+0000 7fd509ffb640 1 -- 172.21.15.16:0/2158080526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5040aca80 con 0x5573018b50a0 2024-09-21T02:07:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.239+0000 7fd509ffb640 1 -- 172.21.15.16:0/2158080526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5040ac6e0 con 0x5573018b50a0 2024-09-21T02:07:31.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.239+0000 7fd520e86740 1 -- 172.21.15.16:0/2158080526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 msgr2=0x557301ca1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.239+0000 7fd520e86740 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd5040aa1e0 tx=0x7fd5040eaf90 comp rx=0 tx=0).stop 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 -- 172.21.15.16:0/2158080526 shutdown_connections 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018ddf60 0x557301c8ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018bd280 0x557301c8f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 --2- 172.21.15.16:0/2158080526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018b50a0 0x557301ca1d50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 -- 172.21.15.16:0/2158080526 >> 172.21.15.16:0/2158080526 conn(0x5573018d2350 msgr2=0x557301c9bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 -- 172.21.15.16:0/2158080526 shutdown_connections 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 -- 172.21.15.16:0/2158080526 wait complete. 2024-09-21T02:07:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 -- start start 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.240+0000 7fd520e86740 1 Processor -- start 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301ca44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301ca9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557301ca1bd0 con 0x5573018bd280 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557301ca1500 con 0x5573018ddf60 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557301ca22a0 con 0x5573018b50a0 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd520e86740 10 client.0 _ll_register_callbacks cb 0x7fd5040eadf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49672/0 (socket says 172.21.15.16:49672) 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 -- 172.21.15.16:0/3740931643 learned_addr learned my addr 172.21.15.16:0/3740931643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd513fff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301ca44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd518eaf640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301ca9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 -- 172.21.15.16:0/3740931643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 msgr2=0x557301ca44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301ca44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 -- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 msgr2=0x557301ca9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301ca9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 -- 172.21.15.16:0/3740931643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557301ca9a60 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd50bfff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc012ab0 tx=0x7fd4fc009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6d9c8a78cb758e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd4f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.241+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc01f020 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fd4fc012660 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5040aca80 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4fc03c050 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 10 client.15717 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557301caace0 con 0x5573018bd280 2024-09-21T02:07:31.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd5040aca80 con 0x5573018bd280 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd4fc0039b0 con 0x5573018bd280 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 10 client.15717 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 10 client.15717 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 20 client.15717 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557301871c90 0x557301d10640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 4 client.15717 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.242+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "status"}) -- 0x557301c90af0 con 0x557301871c90 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd513fff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557301871c90 0x557301d10640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd513fff640 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557301871c90 0x557301d10640 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8a7992f241213c server_cookie=2c28df4a2e1419fc in_seq=0 out_seq=0 2024-09-21T02:07:31.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd4f37fe640 10 client.15717 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:31.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd4f37fe640 1 -- 172.21.15.16:0/3740931643 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557301c90af0 con 0x557301871c90 2024-09-21T02:07:31.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd4f37fe640 10 client.15717 handle_command_reply: tid=0 2024-09-21T02:07:31.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd520e86740 1 client.15717 shutdown 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 msgr2=0x557301ca4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.243+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc012ab0 tx=0x7fd4fc009fb0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 shutdown_connections 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557301871c90 0x557301d10640 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5573018ddf60 0x557301ca9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573018bd280 0x557301ca4a70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 --2- 172.21.15.16:0/3740931643 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5573018b50a0 0x557301ca44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd4f37fe640 0 client.15717 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 >> 172.21.15.16:0/3740931643 conn(0x5573018d2350 msgr2=0x557301c9bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 shutdown_connections 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 1 -- 172.21.15.16:0/3740931643 wait complete. 2024-09-21T02:07:31.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd520e86740 20 client.15717 trim_cache size 0 max 16384 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "8372ca74-77bc-11ef-baf5-efdc52797490", 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24710, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 176.59588663400001, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 28, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 86, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 86, 2024-09-21T02:07:31.249 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 185.79125973800001 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 msgr2=0x7fd5000b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 0x7fd5000b8b80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006010 tx=0x7fd50c0095c0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 msgr2=0x7fd5140ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd50400d950 tx=0x7fd50400de10 comp rx=0 tx=0).stop 2024-09-21T02:07:31.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 shutdown_connections 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fd5000b6670 0x7fd5000b8b80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514108cf0 0x7fd5141000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5141083a0 0x7fd5140ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 --2- 172.21.15.16:0/689642475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd514102a60 0x7fd5140ff670 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 >> 172.21.15.16:0/689642475 conn(0x7fd514079730 msgr2=0x7fd51406b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.244+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 shutdown_connections 2024-09-21T02:07:31.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:31.245+0000 7fd51a912640 1 -- 172.21.15.16:0/689642475 wait complete. 2024-09-21T02:07:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:31 smithi016 ceph-mon[31245]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:07:31.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:31 smithi123 ceph-mon[33453]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:07:31.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:31 smithi161 ceph-mon[34061]: pgmap v363: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-21T02:07:31.959 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-21T02:07:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:33 smithi123 ceph-mon[33453]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:33 smithi016 ceph-mon[31245]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:33.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:33 smithi161 ceph-mon[34061]: pgmap v364: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-21T02:07:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:35 smithi123 ceph-mon[33453]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T02:07:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:35 smithi016 ceph-mon[31245]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T02:07:35.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:35 smithi161 ceph-mon[34061]: pgmap v365: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-21T02:07:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:37 smithi123 ceph-mon[33453]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:37 smithi016 ceph-mon[31245]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:37.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:37 smithi161 ceph-mon[34061]: pgmap v366: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:07:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:07:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:07:39.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: pgmap v367: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-09-21T02:07:39.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:07:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:07:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:41 smithi123 ceph-mon[33453]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:41 smithi016 ceph-mon[31245]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:41.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:41 smithi161 ceph-mon[34061]: pgmap v368: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:43.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:43 smithi123 ceph-mon[33453]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-21T02:07:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:43 smithi016 ceph-mon[31245]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-21T02:07:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:43 smithi161 ceph-mon[34061]: pgmap v369: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-21T02:07:45.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:45 smithi123 ceph-mon[33453]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:45 smithi016 ceph-mon[31245]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:45.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:45 smithi161 ceph-mon[34061]: pgmap v370: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-21T02:07:46.960 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.5 2024-09-21T02:07:46.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:07:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:47.510 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:47 smithi016 ceph-mon[31245]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:47.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:47 smithi123 ceph-mon[33453]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:47 smithi161 ceph-mon[34061]: pgmap v371: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 -- 172.21.15.16:0/3362841700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f01083b0 msgr2=0x7f20f01087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 --2- 172.21.15.16:0/3362841700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f01083b0 0x7f20f01087c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009900 tx=0x7f20dc02a410 comp rx=0 tx=0).stop 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 -- 172.21.15.16:0/3362841700 shutdown_connections 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 --2- 172.21.15.16:0/3362841700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0108d00 0x7f20f010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 --2- 172.21.15.16:0/3362841700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f01083b0 0x7f20f01087c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 --2- 172.21.15.16:0/3362841700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f0102a70 0x7f20f0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 -- 172.21.15.16:0/3362841700 >> 172.21.15.16:0/3362841700 conn(0x7f20f0079710 msgr2=0x7f20f0079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 -- 172.21.15.16:0/3362841700 shutdown_connections 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.752+0000 7f20f7109640 1 -- 172.21.15.16:0/3362841700 wait complete. 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 Processor -- start 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 -- start start 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 0x7f20f019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f01083b0 0x7f20f019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0112870 con 0x7f20f0108d00 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01126f0 con 0x7f20f0102a70 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f7109640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01129f0 con 0x7f20f01083b0 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f56a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f56a6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53762/0 (socket says 172.21.15.16:53762) 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f56a6640 1 -- 172.21.15.16:0/3227674660 learned_addr learned my addr 172.21.15.16:0/3227674660 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.753+0000 7f20f4ea5640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 0x7f20f019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20effff640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f01083b0 0x7f20f019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 -- 172.21.15.16:0/3227674660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f01083b0 msgr2=0x7f20f019b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f01083b0 0x7f20f019b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:47.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 -- 172.21.15.16:0/3227674660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 msgr2=0x7f20f019b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 0x7f20f019b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 -- 172.21.15.16:0/3227674660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f01a6550 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f4ea5640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 0x7f20f019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f56a6640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f20e800cad0 tx=0x7f20e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e800ae20 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f01a6840 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20e8005020 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.754+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e800f6b0 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.755+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f01a6d50 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.756+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f0103cf0 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.757+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f20e80051d0 con 0x7f20f0108d00 2024-09-21T02:07:47.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.758+0000 7f20edffb640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 0x7f20e00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.758+0000 7f20f4ea5640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 0x7f20e00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.758+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f20e80e7b90 con 0x7f20f0108d00 2024-09-21T02:07:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.759+0000 7f20f4ea5640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 0x7f20e00b8b80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20f006c720 tx=0x7f20e400b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.767+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f20e8014030 con 0x7f20f0108d00 2024-09-21T02:07:47.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.997+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20f010a540 con 0x7f20f0108d00 2024-09-21T02:07:47.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:47.997+0000 7f20edffb640 1 -- 172.21.15.16:0/3227674660 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f20e80b2210 con 0x7f20f0108d00 2024-09-21T02:07:47.999 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:07:47.999 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:07:47.999 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 msgr2=0x7f20e00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 0x7f20e00b8b80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20f006c720 tx=0x7f20e400b410 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 msgr2=0x7f20f019bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f20e800cad0 tx=0x7f20e800cf90 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 shutdown_connections 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20e00b6670 0x7f20e00b8b80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20f0108d00 0x7f20f019bcc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f01083b0 0x7f20f019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 --2- 172.21.15.16:0/3227674660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f0102a70 0x7f20f019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:48.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 >> 172.21.15.16:0/3227674660 conn(0x7f20f0079710 msgr2=0x7f20f0104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 shutdown_connections 2024-09-21T02:07:48.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:48.000+0000 7f20f7109640 1 -- 172.21.15.16:0/3227674660 wait complete. 2024-09-21T02:07:48.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:48 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3227674660' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:48.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:48 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3227674660' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:48.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:48 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3227674660' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:07:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:49 smithi123 ceph-mon[33453]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:49 smithi016 ceph-mon[31245]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:49.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:49 smithi161 ceph-mon[34061]: pgmap v372: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:51.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:51 smithi123 ceph-mon[33453]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:07:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:51 smithi016 ceph-mon[31245]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:07:51.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:51 smithi161 ceph-mon[34061]: pgmap v373: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:07:52.158 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0' 2024-09-21T02:07:52.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 134.5 --await-for 45.0 2024-09-21T02:07:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:07:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.064+0000 7fbb290fb640 1 -- 172.21.15.16:0/1583722649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24102040 msgr2=0x7fbb24102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.064+0000 7fbb290fb640 1 --2- 172.21.15.16:0/1583722649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24102040 0x7fbb24102450 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009900 tx=0x7fbb1402a410 comp rx=0 tx=0).stop 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 -- 172.21.15.16:0/1583722649 shutdown_connections 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 --2- 172.21.15.16:0/1583722649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb24110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 --2- 172.21.15.16:0/1583722649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24106290 0x7fbb241066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 --2- 172.21.15.16:0/1583722649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24102040 0x7fbb24102450 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 -- 172.21.15.16:0/1583722649 >> 172.21.15.16:0/1583722649 conn(0x7fbb24079f50 msgr2=0x7fbb2407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 -- 172.21.15.16:0/1583722649 shutdown_connections 2024-09-21T02:07:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.065+0000 7fbb290fb640 1 -- 172.21.15.16:0/1583722649 wait complete. 2024-09-21T02:07:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.066+0000 7fbb290fb640 1 Processor -- start 2024-09-21T02:07:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 -- start start 2024-09-21T02:07:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 0x7fbb2419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 0x7fbb2419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb2419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb24113960 con 0x7fbb24106290 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241137e0 con 0x7fbb24102040 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb290fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb24113ae0 con 0x7fbb24106be0 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 0x7fbb2419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb2259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 0x7fbb2419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb2419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 0x7fbb2419a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:43560/0 (socket says 172.21.15.16:43560) 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb2419fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:35508/0 (socket says 172.21.15.16:35508) 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.067+0000 7fbb22d9d640 1 -- 172.21.15.16:0/4051747621 learned_addr learned my addr 172.21.15.16:0/4051747621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 -- 172.21.15.16:0/4051747621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 msgr2=0x7fbb2419fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb2419fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 -- 172.21.15.16:0/4051747621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 msgr2=0x7fbb2419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 0x7fbb2419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 -- 172.21.15.16:0/4051747621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb241a0340 con 0x7fbb24106290 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb2259c640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 0x7fbb2419aa30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbb24075a40 tx=0x7fbb1800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb0bfff640 1 -- 172.21.15.16:0/4051747621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18007bb0 con 0x7fbb24106290 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb0bfff640 1 -- 172.21.15.16:0/4051747621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb18007d50 con 0x7fbb24106290 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.068+0000 7fbb0bfff640 1 -- 172.21.15.16:0/4051747621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18018660 con 0x7fbb24106290 2024-09-21T02:07:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.069+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb241a0630 con 0x7fbb24106290 2024-09-21T02:07:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.069+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb24102ec0 con 0x7fbb24106290 2024-09-21T02:07:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.072+0000 7fbb2f66f740 1 Processor -- start 2024-09-21T02:07:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.072+0000 7fbb2f66f740 1 -- start start 2024-09-21T02:07:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb0bfff640 1 -- 172.21.15.16:0/4051747621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7fbb1800f450 con 0x7fbb24106290 2024-09-21T02:07:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c92bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c563b60 0x55f52c92c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f52c8a1b90 con 0x55f52c55b980 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f52c890fc0 con 0x55f52c584840 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f52c894dd0 con 0x55f52c563b60 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:43564/0 (socket says 172.21.15.16:43564) 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2359e640 1 -- 172.21.15.16:0/3412348367 learned_addr learned my addr 172.21.15.16:0/3412348367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c92bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.073+0000 7fbb2259c640 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c563b60 0x55f52c92c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.074+0000 7fbb2359e640 1 -- 172.21.15.16:0/3412348367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c563b60 msgr2=0x55f52c92c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.074+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c563b60 0x55f52c92c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.074+0000 7fbb2359e640 1 -- 172.21.15.16:0/3412348367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 msgr2=0x55f52c92bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.074+0000 7fbb0bfff640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 0x7fbb0c0b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.075+0000 7fbb2259c640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 0x7fbb0c0b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.075+0000 7fbb0bfff640 1 -- 172.21.15.16:0/4051747621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fbb180e7950 con 0x7fbb24106290 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.075+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c92bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.075+0000 7fbb2359e640 1 -- 172.21.15.16:0/3412348367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f52c818060 con 0x55f52c584840 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.075+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c92bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.076+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009f70 tx=0x7fbb1c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0101fc28524902 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.076+0000 7fbb2259c640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 0x7fbb0c0b8b30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbb180eafd0 tx=0x7fbb18000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.076+0000 7fbb097fa640 1 -- 172.21.15.16:0/3412348367 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c016070 con 0x55f52c584840 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.076+0000 7fbb097fa640 1 -- 172.21.15.16:0/3412348367 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb1c004d20 con 0x55f52c584840 2024-09-21T02:07:53.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.076+0000 7fbb097fa640 1 -- 172.21.15.16:0/3412348367 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c012760 con 0x55f52c584840 2024-09-21T02:07:53.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3412348367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 msgr2=0x55f52c92b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009f70 tx=0x7fbb1c010800 comp rx=0 tx=0).stop 2024-09-21T02:07:53.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3412348367 shutdown_connections 2024-09-21T02:07:53.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c563b60 0x55f52c92c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c92bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3412348367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c584840 0x55f52c92b600 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3412348367 >> 172.21.15.16:0/3412348367 conn(0x55f52c578990 msgr2=0x55f52c9294f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3412348367 shutdown_connections 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.077+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3412348367 wait complete. 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.078+0000 7fbb2f66f740 1 -- start start 2024-09-21T02:07:53.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.078+0000 7fbb2f66f740 1 Processor -- start 2024-09-21T02:07:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.079+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.079+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9377a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.079+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c584840 0x55f52c94b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.079+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.079+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f52c6f27a0 con 0x55f52c55b980 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53788/0 (socket says 172.21.15.16:53788) 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2359e640 1 -- 172.21.15.16:0/3817194800 learned_addr learned my addr 172.21.15.16:0/3817194800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f52c8360a0 con 0x55f52c563b60 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f52c9425f0 con 0x55f52c584840 2024-09-21T02:07:53.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2f66f740 10 client.0 _ll_register_callbacks cb 0x55f52c8133c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 -- 172.21.15.16:0/3817194800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c584840 msgr2=0x55f52c94b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c584840 0x55f52c94b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 -- 172.21.15.16:0/3817194800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 msgr2=0x55f52c937240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb22d9d640 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52c824880 con 0x55f52c563b60 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.080+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.081+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb1c004d20 con 0x55f52c563b60 2024-09-21T02:07:53.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.081+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9377a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb14000c00 tx=0x7fbb14031c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ae0771e01fdce7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.081+0000 7fbb06ff5640 10 client.25367 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:07:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.081+0000 7fbb06ff5640 1 -- 172.21.15.16:0/3817194800 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14035040 con 0x55f52c563b60 2024-09-21T02:07:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.081+0000 7fbb06ff5640 1 -- 172.21.15.16:0/3817194800 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fbb14004040 con 0x55f52c563b60 2024-09-21T02:07:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb06ff5640 1 -- 172.21.15.16:0/3817194800 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbb14004280 con 0x55f52c563b60 2024-09-21T02:07:53.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 10 client.25367 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55f52c5feb90 con 0x55f52c563b60 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb06ff5640 1 -- 172.21.15.16:0/3817194800 <== mon.2 v2:172.21.15.123:3300/0 4 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fbb140035f0 con 0x55f52c563b60 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 10 client.25367 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 10 client.25367 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 20 client.25367 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c5431c0 0x55f52c951960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 4 client.25367 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.082+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbb1c004d20 con 0x55f52c5431c0 2024-09-21T02:07:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.083+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c5431c0 0x55f52c951960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.083+0000 7fbb06ff5640 10 client.25367 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.083+0000 7fbb2359e640 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c5431c0 0x55f52c951960 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55c4989d64ee149a server_cookie=a541dea579fbc30b in_seq=0 out_seq=0 2024-09-21T02:07:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.084+0000 7fbb06ff5640 1 -- 172.21.15.16:0/3817194800 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7fbb1c004d20 con 0x55f52c5431c0 2024-09-21T02:07:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.084+0000 7fbb06ff5640 10 client.25367 handle_command_reply: tid=0 2024-09-21T02:07:53.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.084+0000 7fbb2f66f740 1 client.25367 shutdown 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.084+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 msgr2=0x55f52c9377a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.084+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9377a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb14000c00 tx=0x7fbb14031c10 comp rx=0 tx=0).stop 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 shutdown_connections 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c5431c0 0x55f52c951960 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c584840 0x55f52c94b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb06ff5640 0 client.25367 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9377a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/3817194800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c937240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 >> 172.21.15.16:0/3817194800 conn(0x55f52c578990 msgr2=0x55f52c92e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 shutdown_connections 2024-09-21T02:07:53.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 1 -- 172.21.15.16:0/3817194800 wait complete. 2024-09-21T02:07:53.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.085+0000 7fbb2f66f740 20 client.25367 trim_cache size 0 max 16384 2024-09-21T02:07:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 Processor -- start 2024-09-21T02:07:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 -- start start 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c5431c0 0x55f52c938c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c934770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241aec10 con 0x55f52c55b980 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241b0750 con 0x55f52c563b60 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.120+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241af2e0 con 0x55f52c5431c0 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53790/0 (socket says 172.21.15.16:53790) 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 -- 172.21.15.16:0/1005858703 learned_addr learned my addr 172.21.15.16:0/1005858703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb2359e640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c5431c0 0x55f52c938c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb2359e640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c934770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 -- 172.21.15.16:0/1005858703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c5431c0 msgr2=0x55f52c938c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c5431c0 0x55f52c938c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 -- 172.21.15.16:0/1005858703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 msgr2=0x55f52c934770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c934770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 -- 172.21.15.16:0/1005858703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f52c92de90 con 0x55f52c55b980 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb2359e640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c934770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:07:53.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbb14001eb0 tx=0x7fbb1403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60beb3dd8d2c0202 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.121+0000 7fbb08ff9640 1 -- 172.21.15.16:0/1005858703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14000e60 con 0x55f52c55b980 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.122+0000 7fbb08ff9640 1 -- 172.21.15.16:0/1005858703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb14003bf0 con 0x55f52c55b980 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.122+0000 7fbb08ff9640 1 -- 172.21.15.16:0/1005858703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14035030 con 0x55f52c55b980 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.122+0000 7fbb2f66f740 1 -- 172.21.15.16:0/1005858703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 msgr2=0x55f52c934230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.122+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbb14001eb0 tx=0x7fbb1403f000 comp rx=0 tx=0).stop 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 -- 172.21.15.16:0/1005858703 shutdown_connections 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c934770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c55b980 0x55f52c934230 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/1005858703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c5431c0 0x55f52c938c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 -- 172.21.15.16:0/1005858703 >> 172.21.15.16:0/1005858703 conn(0x55f52c578990 msgr2=0x55f52c938560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 -- 172.21.15.16:0/1005858703 shutdown_connections 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 -- 172.21.15.16:0/1005858703 wait complete. 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 -- start start 2024-09-21T02:07:53.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.123+0000 7fbb2f66f740 1 Processor -- start 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 0x55f52c953b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9545f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241b0080 con 0x55f52c5431c0 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241af9b0 con 0x55f52c563b60 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbb241acbf0 con 0x55f52c55b980 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2f66f740 10 client.0 _ll_register_callbacks cb 0x7fbb14031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:07:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:35534/0 (socket says 172.21.15.16:35534) 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb22d9d640 1 -- 172.21.15.16:0/2354984766 learned_addr learned my addr 172.21.15.16:0/2354984766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.124+0000 7fbb2359e640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 0x55f52c953b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 -- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 msgr2=0x55f52c9545f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9545f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 -- 172.21.15.16:0/2354984766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 msgr2=0x55f52c953b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 0x55f52c953b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 -- 172.21.15.16:0/2354984766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52c9bc470 con 0x55f52c55b980 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb2359e640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 0x55f52c953b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb22d9d640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb14000c00 tx=0x7fbb14003da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46982ca346d59f20 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.125+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14035070 con 0x55f52c55b980 2024-09-21T02:07:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb14003bf0 con 0x55f52c55b980 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fbb14003ef0 con 0x55f52c55b980 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbb14069050 con 0x55f52c55b980 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb2f66f740 10 client.25441 fetch_fsmap learned FSMap version 28 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f52c9bd6d0 con 0x55f52c55b980 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.126+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbb14003bf0 con 0x55f52c55b980 2024-09-21T02:07:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fbb14063500 con 0x55f52c55b980 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 10 client.25441 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 10 client.25441 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 20 client.25441 populate_metadata read hostname 'smithi016' 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c584840 0x55f52c9bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 4 client.25441 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}] 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 134.5, "await_for": 45.0}) -- 0x55f52c8a3ae0 con 0x55f52c584840 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.127+0000 7fbb2359e640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c584840 0x55f52c9bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.128+0000 7fbb2359e640 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c584840 0x55f52c9bfe10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7922c3ab4e2f1693 server_cookie=4e78a9ffede07b17 in_seq=0 out_seq=0 2024-09-21T02:07:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.128+0000 7fbb06ff5640 10 client.25441 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb06ff5640 1 -- 172.21.15.16:0/2354984766 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55f52c8a3ae0 con 0x55f52c584840 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb06ff5640 10 client.25441 handle_command_reply: tid=0 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb2f66f740 1 client.25441 shutdown 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 msgr2=0x55f52c954070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb14000c00 tx=0x7fbb14003da0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 shutdown_connections 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.138+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f52c584840 0x55f52c9bfe10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f52c563b60 0x55f52c9545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f52c55b980 0x55f52c954070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 --2- 172.21.15.16:0/2354984766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f52c5431c0 0x55f52c953b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb06ff5640 0 client.25441 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 >> 172.21.15.16:0/2354984766 conn(0x55f52c578990 msgr2=0x55f52c92c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 shutdown_connections 2024-09-21T02:07:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 1 -- 172.21.15.16:0/2354984766 wait complete. 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.139+0000 7fbb2f66f740 20 client.25441 trim_cache size 0 max 16384 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 2, 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "d9b03b8f": { 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 2, 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:07:53.141 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 134.5, 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:53.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:53.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:53.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:07:53.143 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 msgr2=0x7fbb0c0b8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 0x7fbb0c0b8b30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbb180eafd0 tx=0x7fbb18000bc0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 msgr2=0x7fbb2419aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 0x7fbb2419aa30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbb24075a40 tx=0x7fbb1800cdd0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 shutdown_connections 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbb0c0b6620 0x7fbb0c0b8b30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbb24106be0 0x7fbb2419fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.140+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb24106290 0x7fbb2419aa30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.141+0000 7fbb290fb640 1 --2- 172.21.15.16:0/4051747621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb24102040 0x7fbb2419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.141+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 >> 172.21.15.16:0/4051747621 conn(0x7fbb24079f50 msgr2=0x7fbb24107620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.141+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 shutdown_connections 2024-09-21T02:07:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:07:53.141+0000 7fbb290fb640 1 -- 172.21.15.16:0/4051747621 wait complete. 2024-09-21T02:07:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:53 smithi016 ceph-mon[31245]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:53.756 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:53 smithi161 ceph-mon[34061]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:53.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:53 smithi123 ceph-mon[33453]: pgmap v374: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:53.975 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd9b03b8f', quiesce took 7.0 seconds. Will release after: 7.5 2024-09-21T02:07:55.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:55 smithi016 ceph-mon[31245]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:07:55.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:55 smithi161 ceph-mon[34061]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:07:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:55 smithi123 ceph-mon[33453]: pgmap v375: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:07:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:57 smithi016 ceph-mon[31245]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:57.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:57 smithi161 ceph-mon[34061]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:57 smithi123 ceph-mon[33453]: pgmap v376: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:07:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:07:59 smithi016 ceph-mon[31245]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:07:59.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:07:59 smithi161 ceph-mon[34061]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:07:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:07:59 smithi123 ceph-mon[33453]: pgmap v377: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:08:00.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:08:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:08:00.154+0000 7f593ed07640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:08:00.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:08:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:08:00.154+0000 7f593ed07640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:08:00.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:08:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:08:00.154+0000 7f593ed07640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:08:00.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:08:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:08:00.155+0000 7f593ed07640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:08:00.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:08:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:08:00.155+0000 7f593ed07640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T02:08:01.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:08:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:02.046 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:01 smithi016 ceph-mon[31245]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T02:08:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:01 smithi161 ceph-mon[34061]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T02:08:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:01 smithi123 ceph-mon[33453]: pgmap v378: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 -- 172.21.15.16:0/2363871519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 msgr2=0x7f091c110660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 --2- 172.21.15.16:0/2363871519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c110660 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0914009ca0 tx=0x7f091402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 -- 172.21.15.16:0/2363871519 shutdown_connections 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 --2- 172.21.15.16:0/2363871519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c110660 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 --2- 172.21.15.16:0/2363871519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 0x7f091c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 --2- 172.21.15.16:0/2363871519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 0x7f091c103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 -- 172.21.15.16:0/2363871519 >> 172.21.15.16:0/2363871519 conn(0x7f091c0ff630 msgr2=0x7f091c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.343+0000 7f09242d1640 1 -- 172.21.15.16:0/2363871519 shutdown_connections 2024-09-21T02:08:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.344+0000 7f09242d1640 1 -- 172.21.15.16:0/2363871519 wait complete. 2024-09-21T02:08:02.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 Processor -- start 2024-09-21T02:08:02.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 -- start start 2024-09-21T02:08:02.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 0x7f091c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:02.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 0x7f091c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:02.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:02.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f091c1139f0 con 0x7f091c109e30 2024-09-21T02:08:02.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f092286e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:02.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f092186c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 0x7f091c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:02.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f092286e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60710/0 (socket says 172.21.15.16:60710) 2024-09-21T02:08:02.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f092286e640 1 -- 172.21.15.16:0/3547583845 learned_addr learned my addr 172.21.15.16:0/3547583845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:02.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092206d640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 0x7f091c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.345+0000 7f09242d1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f091c113870 con 0x7f091c103ba0 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f091c113b70 con 0x7f091c1094e0 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 -- 172.21.15.16:0/3547583845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 msgr2=0x7f091c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 0x7f091c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 -- 172.21.15.16:0/3547583845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 msgr2=0x7f091c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 0x7f091c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091c1a8620 con 0x7f091c109e30 2024-09-21T02:08:02.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.346+0000 7f092286e640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0914005cc0 tx=0x7f0914003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.347+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914050070 con 0x7f091c109e30 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.347+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0914031e10 con 0x7f091c109e30 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.347+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914004800 con 0x7f091c109e30 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.347+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091c19a070 con 0x7f091c109e30 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.347+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091c19a4a0 con 0x7f091c109e30 2024-09-21T02:08:02.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.350+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09000052c0 con 0x7f091c109e30 2024-09-21T02:08:02.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.351+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 154586+0+0 (secure 0 0 0) 0x7f0914053070 con 0x7f091c109e30 2024-09-21T02:08:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.353+0000 7f09137fe640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 0x7f09080c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.353+0000 7f092206d640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 0x7f09080c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:02.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.354+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f091410ffb0 con 0x7f091c109e30 2024-09-21T02:08:02.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.354+0000 7f092206d640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 0x7f09080c9c60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f091c077160 tx=0x7f0918005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:02.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.362+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f091408a710 con 0x7f091c109e30 2024-09-21T02:08:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.800+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f09140d1e70 con 0x7f091c109e30 2024-09-21T02:08:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.818+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09000055b0 con 0x7f091c109e30 2024-09-21T02:08:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.819+0000 7f09137fe640 1 -- 172.21.15.16:0/3547583845 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f09140ca9e0 con 0x7f091c109e30 2024-09-21T02:08:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:08:02.820 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:08:02.821 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:08:02.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.821+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 msgr2=0x7f09080c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.821+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 0x7f09080c9c60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f091c077160 tx=0x7f0918005f90 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.821+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 msgr2=0x7f091c076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.821+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0914005cc0 tx=0x7f0914003ba0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 shutdown_connections 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f09080c7750 0x7f09080c9c60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f091c109e30 0x7f091c076190 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f091c1094e0 0x7f091c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 --2- 172.21.15.16:0/3547583845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f091c103ba0 0x7f091c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 >> 172.21.15.16:0/3547583845 conn(0x7f091c0ff630 msgr2=0x7f091c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:02.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 shutdown_connections 2024-09-21T02:08:02.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:02.822+0000 7f09242d1640 1 -- 172.21.15.16:0/3547583845 wait complete. 2024-09-21T02:08:03.528 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id d9b03b8f --release --await' 2024-09-21T02:08:03.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id d9b03b8f --release --await 2024-09-21T02:08:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:04.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:03 smithi016 ceph-mon[31245]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:08:04.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:03 smithi016 ceph-mon[31245]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T02:08:04.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:03 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3547583845' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:03 smithi161 ceph-mon[34061]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:08:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:03 smithi161 ceph-mon[34061]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T02:08:04.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:03 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3547583845' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:03 smithi123 ceph-mon[33453]: pgmap v379: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-09-21T02:08:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:03 smithi123 ceph-mon[33453]: mgrmap e24: x(active, since 10m), standbys: z, y 2024-09-21T02:08:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:03 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3547583845' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 -- 172.21.15.16:0/1411141797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d641083a0 msgr2=0x7f5d641087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 --2- 172.21.15.16:0/1411141797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d641083a0 0x7f5d641087b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 -- 172.21.15.16:0/1411141797 shutdown_connections 2024-09-21T02:08:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 --2- 172.21.15.16:0/1411141797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64108cf0 0x7f5d6410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 --2- 172.21.15.16:0/1411141797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d641083a0 0x7f5d641087b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 --2- 172.21.15.16:0/1411141797 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d64102a60 0x7f5d64102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 -- 172.21.15.16:0/1411141797 >> 172.21.15.16:0/1411141797 conn(0x7f5d64079730 msgr2=0x7f5d64079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 -- 172.21.15.16:0/1411141797 shutdown_connections 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.424+0000 7f5d6a762640 1 -- 172.21.15.16:0/1411141797 wait complete. 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.425+0000 7f5d6a762640 1 Processor -- start 2024-09-21T02:08:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 -- start start 2024-09-21T02:08:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 0x7f5d6419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d641083a0 0x7f5d6419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64112860 con 0x7f5d64108cf0 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641126e0 con 0x7f5d641083a0 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d6a762640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641129e0 con 0x7f5d64102a60 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d68cff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d68cff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60730/0 (socket says 172.21.15.16:60730) 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d68cff640 1 -- 172.21.15.16:0/3594570938 learned_addr learned my addr 172.21.15.16:0/3594570938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d641083a0 0x7f5d6419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.426+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 0x7f5d6419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 -- 172.21.15.16:0/3594570938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 msgr2=0x7f5d6419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 0x7f5d6419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 -- 172.21.15.16:0/3594570938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d641083a0 msgr2=0x7f5d6419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d641083a0 0x7f5d6419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 -- 172.21.15.16:0/3594570938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d641a0120 con 0x7f5d64108cf0 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 0x7f5d6419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d68cff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00eaa0 tx=0x7f5d5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d61ffb640 1 -- 172.21.15.16:0/3594570938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00cdb0 con 0x7f5d64108cf0 2024-09-21T02:08:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d61ffb640 1 -- 172.21.15.16:0/3594570938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d5c005020 con 0x7f5d64108cf0 2024-09-21T02:08:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.427+0000 7f5d61ffb640 1 -- 172.21.15.16:0/3594570938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0116d0 con 0x7f5d64108cf0 2024-09-21T02:08:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.428+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d641a0410 con 0x7f5d64108cf0 2024-09-21T02:08:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.428+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d64113000 con 0x7f5d64108cf0 2024-09-21T02:08:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.432+0000 7f5d70cd6740 1 Processor -- start 2024-09-21T02:08:04.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.433+0000 7f5d61ffb640 1 -- 172.21.15.16:0/3594570938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5d5c0051d0 con 0x7f5d64108cf0 2024-09-21T02:08:04.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.433+0000 7f5d70cd6740 1 -- start start 2024-09-21T02:08:04.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.433+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd719a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60740/0 (socket says 172.21.15.16:60740) 2024-09-21T02:08:04.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd352840 0x5607fd719f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607fd604dc0 con 0x5607fd373520 2024-09-21T02:08:04.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5607fd6911f0 con 0x5607fd352840 2024-09-21T02:08:04.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5607fd680620 con 0x5607fd34a660 2024-09-21T02:08:04.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 -- 172.21.15.16:0/439237695 learned_addr learned my addr 172.21.15.16:0/439237695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:04.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d63fff640 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd719a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 -- 172.21.15.16:0/439237695 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 msgr2=0x5607fd719a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd719a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 -- 172.21.15.16:0/439237695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd352840 msgr2=0x5607fd719f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd352840 0x5607fd719f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d5bfff640 1 -- 172.21.15.16:0/439237695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607fd648fd0 con 0x5607fd373520 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.434+0000 7f5d61ffb640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 0x7f5d500b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.435+0000 7f5d61ffb640 1 -- 172.21.15.16:0/3594570938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f5d5c0e7f20 con 0x7f5d64108cf0 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.435+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 0x7f5d500b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.435+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65846ef59d1163d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.436+0000 7f5d5a7fc640 1 -- 172.21.15.16:0/439237695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c035040 con 0x5607fd373520 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.436+0000 7f5d5a7fc640 1 -- 172.21.15.16:0/439237695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d4c0042f0 con 0x5607fd373520 2024-09-21T02:08:04.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.436+0000 7f5d5a7fc640 1 -- 172.21.15.16:0/439237695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c003a10 con 0x5607fd373520 2024-09-21T02:08:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 -- 172.21.15.16:0/439237695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 msgr2=0x5607fd719510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c000bc0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 0x7f5d500b8c90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5607fd718190 tx=0x7f5d540094a0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 -- 172.21.15.16:0/439237695 shutdown_connections 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd352840 0x5607fd719f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd719a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/439237695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd373520 0x5607fd719510 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 -- 172.21.15.16:0/439237695 >> 172.21.15.16:0/439237695 conn(0x5607fd367910 msgr2=0x5607fd717400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 -- 172.21.15.16:0/439237695 shutdown_connections 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.437+0000 7f5d70cd6740 1 -- 172.21.15.16:0/439237695 wait complete. 2024-09-21T02:08:04.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.438+0000 7f5d70cd6740 1 -- start start 2024-09-21T02:08:04.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.438+0000 7f5d70cd6740 1 Processor -- start 2024-09-21T02:08:04.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.439+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.439+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.439+0000 7f5d63fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:57966/0 (socket says 172.21.15.16:57966) 2024-09-21T02:08:04.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60752/0 (socket says 172.21.15.16:60752) 2024-09-21T02:08:04.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 -- 172.21.15.16:0/573173049 learned_addr learned my addr 172.21.15.16:0/573173049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd739570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607fd61b300 con 0x5607fd352840 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5607fd684430 con 0x5607fd331ea0 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5607fd4e1bb0 con 0x5607fd34a660 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d68cff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd739570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 -- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 msgr2=0x5607fd739570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd739570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 -- 172.21.15.16:0/573173049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 msgr2=0x5607fd72da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d70cd6740 10 client.0 _ll_register_callbacks cb 0x5607fd635120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.440+0000 7f5d63fff640 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607fd60a780 con 0x5607fd34a660 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d4c0042f0 con 0x5607fd34a660 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:08:04.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d63fff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d540126d0 tx=0x7f5d540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5b4ab307a043dd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d437fe640 10 client.25462 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d437fe640 1 -- 172.21.15.16:0/573173049 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5400e920 con 0x5607fd34a660 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.441+0000 7f5d437fe640 1 -- 172.21.15.16:0/573173049 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f5d5400eac0 con 0x5607fd34a660 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d437fe640 1 -- 172.21.15.16:0/573173049 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5d5400ed20 con 0x5607fd34a660 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 10 client.25462 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5607fd3deb00 con 0x5607fd34a660 2024-09-21T02:08:04.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d437fe640 1 -- 172.21.15.16:0/573173049 <== mon.1 v2:172.21.15.161:3300/0 4 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f5d5400f040 con 0x5607fd34a660 2024-09-21T02:08:04.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 10 client.25462 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:04.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 10 client.25462 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 20 client.25462 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.442+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd307250 0x5607fd73f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.443+0000 7f5d70cd6740 4 client.25462 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.443+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607fd693140 con 0x5607fd307250 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.443+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd307250 0x5607fd73f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.443+0000 7f5d437fe640 10 client.25462 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.443+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd307250 0x5607fd73f610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=260049bea1d276d8 server_cookie=8088631af225848b in_seq=0 out_seq=0 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.444+0000 7f5d437fe640 1 -- 172.21.15.16:0/573173049 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5607fd693140 con 0x5607fd307250 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.444+0000 7f5d437fe640 10 client.25462 handle_command_reply: tid=0 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.444+0000 7f5d70cd6740 1 client.25462 shutdown 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.444+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 msgr2=0x5607fd72d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.444+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d540126d0 tx=0x7f5d540046c0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 shutdown_connections 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd307250 0x5607fd73f610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d437fe640 0 client.25462 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:04.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd739570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd352840 0x5607fd72da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/573173049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd72d4f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 >> 172.21.15.16:0/573173049 conn(0x5607fd367910 msgr2=0x5607fd71c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 shutdown_connections 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 1 -- 172.21.15.16:0/573173049 wait complete. 2024-09-21T02:08:04.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.445+0000 7f5d70cd6740 20 client.25462 trim_cache size 0 max 16384 2024-09-21T02:08:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.476+0000 7f5d70cd6740 1 Processor -- start 2024-09-21T02:08:04.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.476+0000 7f5d70cd6740 1 -- start start 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.476+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 0x5607fd71cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd71af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641aeaf0 con 0x5607fd307250 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641b0630 con 0x5607fd331ea0 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641af1c0 con 0x5607fd34a660 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d68cff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd71af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d68cff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:57982/0 (socket says 172.21.15.16:57982) 2024-09-21T02:08:04.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d68cff640 1 -- 172.21.15.16:0/593861247 learned_addr learned my addr 172.21.15.16:0/593861247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:04.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d63fff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 0x5607fd71cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d5bfff640 1 -- 172.21.15.16:0/593861247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 msgr2=0x5607fd71e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.477+0000 7f5d5bfff640 1 -- 172.21.15.16:0/593861247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 msgr2=0x5607fd71cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 0x5607fd71cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d5bfff640 1 -- 172.21.15.16:0/593861247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607fd455960 con 0x5607fd331ea0 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d68cff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d63fff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 0x5607fd71cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd71af00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004000 tx=0x7f5d4c0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddfa9882667e5168 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d59ffb640 1 -- 172.21.15.16:0/593861247 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c035040 con 0x5607fd331ea0 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d59ffb640 1 -- 172.21.15.16:0/593861247 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d4c032bd0 con 0x5607fd331ea0 2024-09-21T02:08:04.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.478+0000 7f5d59ffb640 1 -- 172.21.15.16:0/593861247 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c03c6a0 con 0x5607fd331ea0 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 -- 172.21.15.16:0/593861247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 msgr2=0x5607fd71af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd71af00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004000 tx=0x7f5d4c0035f0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 -- 172.21.15.16:0/593861247 shutdown_connections 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd71e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd331ea0 0x5607fd71af00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/593861247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd307250 0x5607fd71cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.479+0000 7f5d70cd6740 1 -- 172.21.15.16:0/593861247 >> 172.21.15.16:0/593861247 conn(0x5607fd367910 msgr2=0x5607fd72f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.480+0000 7f5d70cd6740 1 -- 172.21.15.16:0/593861247 shutdown_connections 2024-09-21T02:08:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.480+0000 7f5d70cd6740 1 -- 172.21.15.16:0/593861247 wait complete. 2024-09-21T02:08:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.480+0000 7f5d70cd6740 1 -- start start 2024-09-21T02:08:04.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.480+0000 7f5d70cd6740 1 Processor -- start 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd307250 0x5607fd731e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd7a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641aff60 con 0x5607fd331ea0 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641af890 con 0x5607fd307250 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641acad0 con 0x5607fd34a660 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d70cd6740 10 client.0 _ll_register_callbacks cb 0x7f5d54009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60778/0 (socket says 172.21.15.16:60778) 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 -- 172.21.15.16:0/3413085858 learned_addr learned my addr 172.21.15.16:0/3413085858 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d68cff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd7a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 -- 172.21.15.16:0/3413085858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 msgr2=0x5607fd7a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd7a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 -- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd307250 msgr2=0x5607fd731e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.481+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd307250 0x5607fd731e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d5bfff640 1 -- 172.21.15.16:0/3413085858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607fd7a3680 con 0x5607fd331ea0 2024-09-21T02:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d5bfff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009850 tx=0x7f5d4c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27724161de7c7671 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d437fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c035040 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d4c032bd0 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.482+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f5d4c003660 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.483+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5d4c003e20 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.483+0000 7f5d70cd6740 10 client.15768 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.483+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5607fd7aa950 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.483+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d4c032bd0 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.483+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f5d4c04c040 con 0x5607fd331ea0 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 10 client.15768 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 10 client.15768 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 20 client.15768 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd352840 0x5607fd7acc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 4 client.15768 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d9b03b8f", "release": true, "await": true}] 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d9b03b8f", "release": true, "await": true}) -- 0x5607fd68ebf0 con 0x5607fd352840 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd352840 0x5607fd7acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d437fe640 10 client.15768 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.484+0000 7f5d63fff640 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd352840 0x5607fd7acc30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94592e7d84940c7d server_cookie=d1c4e74d654aec90 in_seq=0 out_seq=0 2024-09-21T02:08:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d437fe640 1 -- 172.21.15.16:0/3413085858 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5607fd68ebf0 con 0x5607fd352840 2024-09-21T02:08:04.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d437fe640 10 client.15768 handle_command_reply: tid=0 2024-09-21T02:08:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 client.15768 shutdown 2024-09-21T02:08:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 msgr2=0x5607fd732430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009850 tx=0x7f5d4c005e80 comp rx=0 tx=0).stop 2024-09-21T02:08:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 shutdown_connections 2024-09-21T02:08:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5607fd352840 0x5607fd7acc30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d437fe640 0 client.15768 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5607fd34a660 0x5607fd7a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607fd331ea0 0x5607fd732430 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 --2- 172.21.15.16:0/3413085858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5607fd307250 0x5607fd731e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 >> 172.21.15.16:0/3413085858 conn(0x5607fd367910 msgr2=0x5607fd72f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 shutdown_connections 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 1 -- 172.21.15.16:0/3413085858 wait complete. 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.487+0000 7f5d70cd6740 20 client.15768 trim_cache size 0 max 16384 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:08:04.489 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 4, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "d9b03b8f": { 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 4, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 134.5, 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:08:04.490 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:04.491 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.488+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 msgr2=0x7f5d500b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.488+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 0x7f5d500b8c90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5607fd718190 tx=0x7f5d540094a0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.488+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 msgr2=0x7f5d6419f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.488+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00eaa0 tx=0x7f5d5c00ef60 comp rx=0 tx=0).stop 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 shutdown_connections 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f5d500b6780 0x7f5d500b8c90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d64108cf0 0x7f5d6419f9f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d641083a0 0x7f5d6419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 --2- 172.21.15.16:0/3594570938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d64102a60 0x7f5d6419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 >> 172.21.15.16:0/3594570938 conn(0x7f5d64079730 msgr2=0x7f5d6406b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 shutdown_connections 2024-09-21T02:08:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:04.489+0000 7f5d6a762640 1 -- 172.21.15.16:0/3594570938 wait complete. 2024-09-21T02:08:05.220 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd9b03b8f', total seconds elapsed: 18.3 2024-09-21T02:08:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:05 smithi016 ceph-mon[31245]: pgmap v380: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-09-21T02:08:06.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:05 smithi161 ceph-mon[34061]: pgmap v380: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-09-21T02:08:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:05 smithi123 ceph-mon[33453]: pgmap v380: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-09-21T02:08:08.072 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:07 smithi123 ceph-mon[33453]: pgmap v381: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:07 smithi016 ceph-mon[31245]: pgmap v381: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:08.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:07 smithi161 ceph-mon[34061]: pgmap v381: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:09 smithi016 ceph-mon[31245]: pgmap v382: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:10.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:09 smithi161 ceph-mon[34061]: pgmap v382: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:09 smithi123 ceph-mon[33453]: pgmap v382: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:11 smithi016 ceph-mon[31245]: pgmap v383: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:12.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:11 smithi161 ceph-mon[34061]: pgmap v383: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:11 smithi123 ceph-mon[33453]: pgmap v383: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:13 smithi016 ceph-mon[31245]: pgmap v384: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:14.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:13 smithi161 ceph-mon[34061]: pgmap v384: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:13 smithi123 ceph-mon[33453]: pgmap v384: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-09-21T02:08:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:15 smithi016 ceph-mon[31245]: pgmap v385: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:16.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:15 smithi161 ceph-mon[34061]: pgmap v385: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:15 smithi123 ceph-mon[33453]: pgmap v385: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-09-21T02:08:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:17 smithi016 ceph-mon[31245]: pgmap v386: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:17 smithi161 ceph-mon[34061]: pgmap v386: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:17 smithi123 ceph-mon[33453]: pgmap v386: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:19 smithi016 ceph-mon[31245]: pgmap v387: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:20.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:19 smithi161 ceph-mon[34061]: pgmap v387: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:19 smithi123 ceph-mon[33453]: pgmap v387: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-21T02:08:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:21 smithi016 ceph-mon[31245]: pgmap v388: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T02:08:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:21 smithi161 ceph-mon[34061]: pgmap v388: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T02:08:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:21 smithi123 ceph-mon[33453]: pgmap v388: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-21T02:08:24.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:23 smithi016 ceph-mon[31245]: pgmap v389: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:24.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:23 smithi161 ceph-mon[34061]: pgmap v389: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:23 smithi123 ceph-mon[33453]: pgmap v389: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:24.461 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 92% (11725/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-09-21T02:08:25.192 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:25.193 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T02:08:25.196 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T02:08:25.233 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T02:08:25.235 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T02:08:25.235 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T02:08:25.995 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T02:08:26.068 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T02:08:26.109 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T02:08:26.110 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T02:08:26.110 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T02:08:26.150 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T02:08:26.153 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:08:26.153 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T02:08:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:25 smithi016 ceph-mon[31245]: pgmap v390: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:08:26.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:25 smithi161 ceph-mon[34061]: pgmap v390: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:08:26.173 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-09-21T02:08:26.175 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-09-21T02:08:26.175 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-09-21T02:08:26.246 INFO:tasks.workunit.client.0.smithi016.stdout:getting iogen 2024-09-21T02:08:26.247 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'getting iogen' 2024-09-21T02:08:26.247 INFO:tasks.workunit.client.0.smithi016.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-09-21T02:08:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:25 smithi123 ceph-mon[33453]: pgmap v390: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:08:26.272 INFO:tasks.workunit.client.0.smithi016.stderr:--2024-09-21 02:08:26-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-09-21T02:08:26.272 INFO:tasks.workunit.client.0.smithi016.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-09-21T02:08:26.294 INFO:tasks.workunit.client.0.smithi016.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-09-21T02:08:26.315 INFO:tasks.workunit.client.0.smithi016.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-21T02:08:26.315 INFO:tasks.workunit.client.0.smithi016.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-09-21T02:08:26.317 INFO:tasks.workunit.client.0.smithi016.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-09-21T02:08:26.321 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:26.321 INFO:tasks.workunit.client.0.smithi016.stderr: 0K ....... 100% 1.29G=0s 2024-09-21T02:08:26.322 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:26.322 INFO:tasks.workunit.client.0.smithi016.stderr:2024-09-21 02:08:26 (1.29 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-09-21T02:08:26.322 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-09-21T02:08:26.322 INFO:tasks.workunit.client.0.smithi016.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-09-21T02:08:26.339 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0 2024-09-21T02:08:26.341 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src 2024-09-21T02:08:26.343 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src/iogen.8 2024-09-21T02:08:26.345 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src/iogen.c 2024-09-21T02:08:26.348 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/README 2024-09-21T02:08:26.351 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/Makefile.bsd 2024-09-21T02:08:26.353 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/Makefile 2024-09-21T02:08:26.356 INFO:tasks.workunit.client.0.smithi016.stderr:+ cd iogen_3.1p0 2024-09-21T02:08:26.358 INFO:tasks.workunit.client.0.smithi016.stdout:making iogen 2024-09-21T02:08:26.358 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'making iogen' 2024-09-21T02:08:26.358 INFO:tasks.workunit.client.0.smithi016.stderr:+ make 2024-09-21T02:08:26.369 INFO:tasks.workunit.client.0.smithi016.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-09-21T02:08:26.599 INFO:tasks.workunit.client.0.smithi016.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-09-21T02:08:26.695 INFO:tasks.workunit.client.0.smithi016.stdout:running iogen 2024-09-21T02:08:26.695 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'running iogen' 2024-09-21T02:08:26.695 INFO:tasks.workunit.client.0.smithi016.stderr:+ ./iogen -n 5 -s 2g 2024-09-21T02:08:26.696 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 90444 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 90445 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 90446 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 90447 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 90448 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'sleep for 10 min' 2024-09-21T02:08:26.697 INFO:tasks.workunit.client.0.smithi016.stdout:sleep for 10 min 2024-09-21T02:08:26.698 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 600 2024-09-21T02:08:28.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:27 smithi016 ceph-mon[31245]: pgmap v391: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:28.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:27 smithi161 ceph-mon[34061]: pgmap v391: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:27 smithi123 ceph-mon[33453]: pgmap v391: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:08:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:29 smithi016 ceph-mon[31245]: pgmap v392: 129 pgs: 129 active+clean; 25 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T02:08:30.179 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:29 smithi161 ceph-mon[34061]: pgmap v392: 129 pgs: 129 active+clean; 25 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T02:08:30.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:29 smithi123 ceph-mon[33453]: pgmap v392: 129 pgs: 129 active+clean; 25 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T02:08:32.149 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.0 2024-09-21T02:08:32.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:08:32.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:31 smithi016 ceph-mon[31245]: pgmap v393: 129 pgs: 129 active+clean; 164 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 21 op/s 2024-09-21T02:08:32.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:31 smithi161 ceph-mon[34061]: pgmap v393: 129 pgs: 129 active+clean; 164 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 21 op/s 2024-09-21T02:08:32.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:31 smithi123 ceph-mon[33453]: pgmap v393: 129 pgs: 129 active+clean; 164 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 21 op/s 2024-09-21T02:08:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 -- 172.21.15.16:0/3164104166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 msgr2=0x7fa9f0103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 --2- 172.21.15.16:0/3164104166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0103f90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009900 tx=0x7fa9e002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 -- 172.21.15.16:0/3164104166 shutdown_connections 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 --2- 172.21.15.16:0/3164104166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 0x7fa9f0110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 --2- 172.21.15.16:0/3164104166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 --2- 172.21.15.16:0/3164104166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0103f90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 -- 172.21.15.16:0/3164104166 >> 172.21.15.16:0/3164104166 conn(0x7fa9f00ff610 msgr2=0x7fa9f0101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 -- 172.21.15.16:0/3164104166 shutdown_connections 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.217+0000 7fa9f6087640 1 -- 172.21.15.16:0/3164104166 wait complete. 2024-09-21T02:08:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 Processor -- start 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 -- start start 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 0x7fa9f0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0113980 con 0x7fa9f0103b80 2024-09-21T02:08:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0113800 con 0x7fa9f01094c0 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.218+0000 7fa9f6087640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f0113b00 con 0x7fa9f0109e10 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9effff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 0x7fa9f0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:46190/0 (socket says 172.21.15.16:46190) 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 -- 172.21.15.16:0/2170996107 learned_addr learned my addr 172.21.15.16:0/2170996107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9ef7fe640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 -- 172.21.15.16:0/2170996107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 msgr2=0x7fa9f0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 0x7fa9f0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 -- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 msgr2=0x7fa9f0075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.219+0000 7fa9eeffd640 1 -- 172.21.15.16:0/2170996107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f01a86b0 con 0x7fa9f01094c0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9eeffd640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e9f0 tx=0x7fa9e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400ccc0 con 0x7fa9f01094c0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f01a89a0 con 0x7fa9f01094c0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9e40108c0 con 0x7fa9f01094c0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f01a8f30 con 0x7fa9f01094c0 2024-09-21T02:08:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.220+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4018660 con 0x7fa9f01094c0 2024-09-21T02:08:33.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.221+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9dc0052c0 con 0x7fa9f01094c0 2024-09-21T02:08:33.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.223+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa9e4010430 con 0x7fa9f01094c0 2024-09-21T02:08:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.224+0000 7fa9ecff9640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 0x7fa9d80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.224+0000 7fa9ef7fe640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 0x7fa9d80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:33.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.224+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fa9e40e8060 con 0x7fa9f01094c0 2024-09-21T02:08:33.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.225+0000 7fa9ef7fe640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 0x7fa9d80b8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0077160 tx=0x7fa9e00058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.227+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa9e40b2620 con 0x7fa9f01094c0 2024-09-21T02:08:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.616+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9dc0055b0 con 0x7fa9f01094c0 2024-09-21T02:08:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.617+0000 7fa9ecff9640 1 -- 172.21.15.16:0/2170996107 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fa9e40b7560 con 0x7fa9f01094c0 2024-09-21T02:08:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:08:33.618 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:08:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 msgr2=0x7fa9d80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 0x7fa9d80b8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0077160 tx=0x7fa9e00058d0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 msgr2=0x7fa9f0075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e9f0 tx=0x7fa9e400eeb0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 shutdown_connections 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa9d80b6700 0x7fa9d80b8c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa9f0109e10 0x7fa9f0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9f01094c0 0x7fa9f0075c50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 --2- 172.21.15.16:0/2170996107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9f0103b80 0x7fa9f0075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 >> 172.21.15.16:0/2170996107 conn(0x7fa9f00ff610 msgr2=0x7fa9f0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 shutdown_connections 2024-09-21T02:08:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:33.620+0000 7fa9f6087640 1 -- 172.21.15.16:0/2170996107 wait complete. 2024-09-21T02:08:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:33 smithi016 ceph-mon[31245]: pgmap v394: 129 pgs: 129 active+clean; 432 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 MiB/s wr, 32 op/s 2024-09-21T02:08:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:33 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2170996107' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:33 smithi161 ceph-mon[34061]: pgmap v394: 129 pgs: 129 active+clean; 432 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 MiB/s wr, 32 op/s 2024-09-21T02:08:34.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:33 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2170996107' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:34.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:33 smithi123 ceph-mon[33453]: pgmap v394: 129 pgs: 129 active+clean; 432 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 36 MiB/s wr, 32 op/s 2024-09-21T02:08:34.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:33 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2170996107' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:34.614 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0' 2024-09-21T02:08:34.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 137.0 --await-for 45.0 2024-09-21T02:08:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 -- 172.21.15.16:0/947595313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102fa0 msgr2=0x7f111c10fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 --2- 172.21.15.16:0/947595313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102fa0 0x7f111c10fcc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f110c009860 tx=0x7f110c02a3d0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 -- 172.21.15.16:0/947595313 shutdown_connections 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 --2- 172.21.15.16:0/947595313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c110200 0x7f111c112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 --2- 172.21.15.16:0/947595313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102fa0 0x7f111c10fcc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 --2- 172.21.15.16:0/947595313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f111c102650 0x7f111c102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 -- 172.21.15.16:0/947595313 >> 172.21.15.16:0/947595313 conn(0x7f111c079760 msgr2=0x7f111c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.663+0000 7f112335f640 1 -- 172.21.15.16:0/947595313 shutdown_connections 2024-09-21T02:08:35.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.664+0000 7f112335f640 1 -- 172.21.15.16:0/947595313 wait complete. 2024-09-21T02:08:35.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.664+0000 7f112335f640 1 Processor -- start 2024-09-21T02:08:35.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 -- start start 2024-09-21T02:08:35.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102650 0x7f111c19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f111c110200 0x7f111c1a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f111c115b20 con 0x7f111c102fa0 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f111c1159a0 con 0x7f111c102650 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f112335f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f111c115ca0 con 0x7f111c110200 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f11208fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f11208fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44478/0 (socket says 172.21.15.16:44478) 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.665+0000 7f11208fa640 1 -- 172.21.15.16:0/25429960 learned_addr learned my addr 172.21.15.16:0/25429960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11210fb640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102650 0x7f111c19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 -- 172.21.15.16:0/25429960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f111c110200 msgr2=0x7f111c1a1e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f111c110200 0x7f111c1a1e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 -- 172.21.15.16:0/25429960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102650 msgr2=0x7f111c19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102650 0x7f111c19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 -- 172.21.15.16:0/25429960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c1a2560 con 0x7f111c102fa0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.666+0000 7f11208fa640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f110c000c00 tx=0x7f110c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.667+0000 7f11127fc640 1 -- 172.21.15.16:0/25429960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c040070 con 0x7f111c102fa0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.667+0000 7f11127fc640 1 -- 172.21.15.16:0/25429960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f110c004b50 con 0x7f111c102fa0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.667+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111c1a27f0 con 0x7f111c102fa0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.667+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111c1a8fb0 con 0x7f111c102fa0 2024-09-21T02:08:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.667+0000 7f11127fc640 1 -- 172.21.15.16:0/25429960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c030b40 con 0x7f111c102fa0 2024-09-21T02:08:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.670+0000 7f11127fc640 1 -- 172.21.15.16:0/25429960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f110c039040 con 0x7f111c102fa0 2024-09-21T02:08:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.671+0000 7f11298d3740 1 Processor -- start 2024-09-21T02:08:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.671+0000 7f11298d3740 1 -- start start 2024-09-21T02:08:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.671+0000 7f11298d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.671+0000 7f11298d3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.671+0000 7f11298d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e8b60 0x5648717b9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11298d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564871726b90 con 0x5648713e8b60 2024-09-21T02:08:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11298d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564871715fc0 con 0x564871409840 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11298d3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564871719dd0 con 0x5648713e0980 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:46212/0 (socket says 172.21.15.16:46212) 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 -- 172.21.15.16:0/3709844367 learned_addr learned my addr 172.21.15.16:0/3709844367 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11210fb640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11208fa640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e8b60 0x5648717b9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 -- 172.21.15.16:0/3709844367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 msgr2=0x5648717b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 -- 172.21.15.16:0/3709844367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e8b60 msgr2=0x5648717b9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e8b60 0x5648717b9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11218fc640 1 -- 172.21.15.16:0/3709844367 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487169d060 con 0x564871409840 2024-09-21T02:08:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.672+0000 7f11210fb640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:08:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.673+0000 7f11218fc640 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1114009f80 tx=0x7f111400e880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bff5a3da315b59 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.673+0000 7f11127fc640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 0x7f11080b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.673+0000 7f11210fb640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 0x7f11080b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.673+0000 7f11077fe640 1 -- 172.21.15.16:0/3709844367 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114004d20 con 0x564871409840 2024-09-21T02:08:35.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.674+0000 7f11077fe640 1 -- 172.21.15.16:0/3709844367 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1114003e00 con 0x564871409840 2024-09-21T02:08:35.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.674+0000 7f11127fc640 1 -- 172.21.15.16:0/25429960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f110c108bc0 con 0x7f111c102fa0 2024-09-21T02:08:35.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.674+0000 7f11077fe640 1 -- 172.21.15.16:0/3709844367 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111400c640 con 0x564871409840 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11210fb640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 0x7f11080b8ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5648717b7a90 tx=0x7f1118005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 -- 172.21.15.16:0/3709844367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 msgr2=0x5648717b8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1114009f80 tx=0x7f111400e880 comp rx=0 tx=0).stop 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 -- 172.21.15.16:0/3709844367 shutdown_connections 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e8b60 0x5648717b9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 --2- 172.21.15.16:0/3709844367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564871409840 0x5648717b8e10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.675+0000 7f11298d3740 1 -- 172.21.15.16:0/3709844367 >> 172.21.15.16:0/3709844367 conn(0x5648713fd990 msgr2=0x5648717b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.676+0000 7f11298d3740 1 -- 172.21.15.16:0/3709844367 shutdown_connections 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.676+0000 7f11298d3740 1 -- 172.21.15.16:0/3709844367 wait complete. 2024-09-21T02:08:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.676+0000 7f11298d3740 1 -- start start 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.676+0000 7f11298d3740 1 Processor -- start 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 0x5648717c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564871409840 0x5648717d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5648715777a0 con 0x564871409840 2024-09-21T02:08:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5648716bb0a0 con 0x5648713e0980 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5648717c7640 con 0x5648713e8b60 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11298d3740 10 client.0 _ll_register_callbacks cb 0x5648716983c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11208fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11210fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 0x5648717c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11208fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:46432/0 (socket says 172.21.15.16:46432) 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11208fa640 1 -- 172.21.15.16:0/4195427419 learned_addr learned my addr 172.21.15.16:0/4195427419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.677+0000 7f11218fc640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564871409840 0x5648717d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 -- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 msgr2=0x5648717c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 0x5648717c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 -- 172.21.15.16:0/4195427419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564871409840 msgr2=0x5648717d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564871409840 0x5648717d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 -- 172.21.15.16:0/4195427419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648716a9880 con 0x5648713e8b60 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11210fb640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 0x5648717c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:08:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11208fa640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f110c031000 tx=0x7f110c02a3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1592e7fc695620c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f1104ff9640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c0ca540 con 0x5648713e8b60 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1114003e00 con 0x5648713e8b60 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f110c0d7c30 con 0x5648713e8b60 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f110c0cde10 con 0x5648713e8b60 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11298d3740 10 client.25483 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:35.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.678+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564871483b90 con 0x5648713e8b60 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1114003e00 con 0x5648713e8b60 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f110c0cd900 con 0x5648713e8b60 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 10 client.25483 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 10 client.25483 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 20 client.25483 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5648713c81c0 0x5648717d6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 4 client.25483 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564871728ae0 con 0x5648713c81c0 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.679+0000 7f11210fb640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5648713c81c0 0x5648717d6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.680+0000 7f1104ff9640 10 client.25483 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.680+0000 7f11210fb640 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5648713c81c0 0x5648717d6a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=587d93b0d6e1a588 server_cookie=27e60665bdc47610 in_seq=0 out_seq=0 2024-09-21T02:08:35.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f1104ff9640 1 -- 172.21.15.16:0/4195427419 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564871728ae0 con 0x5648713c81c0 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f1104ff9640 10 client.25483 handle_command_reply: tid=0 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 client.25483 shutdown 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 msgr2=0x5648717c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f110c031000 tx=0x7f110c02a3d0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 shutdown_connections 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5648713c81c0 0x5648717d6a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f1104ff9640 0 client.25483 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564871409840 0x5648717d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e8b60 0x5648717c4f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 --2- 172.21.15.16:0/4195427419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e0980 0x5648717c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 >> 172.21.15.16:0/4195427419 conn(0x5648713fd990 msgr2=0x5648717bbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 shutdown_connections 2024-09-21T02:08:35.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.681+0000 7f11298d3740 1 -- 172.21.15.16:0/4195427419 wait complete. 2024-09-21T02:08:35.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.682+0000 7f11298d3740 20 client.25483 trim_cache size 0 max 16384 2024-09-21T02:08:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.709+0000 7f11298d3740 1 Processor -- start 2024-09-21T02:08:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.709+0000 7f11298d3740 1 -- start start 2024-09-21T02:08:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.709+0000 7f11298d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11298d3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11298d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x5648717c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11298d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f110000feb0 con 0x5648713c81c0 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11298d3740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11000119f0 con 0x5648713e8b60 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11298d3740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1100010580 con 0x5648713e0980 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44500/0 (socket says 172.21.15.16:44500) 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 -- 172.21.15.16:0/1701641973 learned_addr learned my addr 172.21.15.16:0/1701641973 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11218fc640 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x5648717c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11208fa640 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 -- 172.21.15.16:0/1701641973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 msgr2=0x5648717b6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 -- 172.21.15.16:0/1701641973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 msgr2=0x5648717c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x5648717c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.710+0000 7f11210fb640 1 -- 172.21.15.16:0/1701641973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648717bbc70 con 0x5648713c81c0 2024-09-21T02:08:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.711+0000 7f11210fb640 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x5648717b7cb0 tx=0x7f111801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fb8d61c013d0bb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.711+0000 7f1106ffd640 1 -- 172.21.15.16:0/1701641973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111800f580 con 0x5648713c81c0 2024-09-21T02:08:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.711+0000 7f1106ffd640 1 -- 172.21.15.16:0/1701641973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1118038070 con 0x5648713c81c0 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.711+0000 7f1106ffd640 1 -- 172.21.15.16:0/1701641973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118033a40 con 0x5648713c81c0 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 -- 172.21.15.16:0/1701641973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 msgr2=0x5648717ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x5648717b7cb0 tx=0x7f111801b000 comp rx=0 tx=0).stop 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 -- 172.21.15.16:0/1701641973 shutdown_connections 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x5648717c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713e0980 0x5648717b6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 --2- 172.21.15.16:0/1701641973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713c81c0 0x5648717ad450 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 -- 172.21.15.16:0/1701641973 >> 172.21.15.16:0/1701641973 conn(0x5648713fd990 msgr2=0x5648717c5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.712+0000 7f11298d3740 1 -- 172.21.15.16:0/1701641973 shutdown_connections 2024-09-21T02:08:35.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.713+0000 7f11298d3740 1 -- 172.21.15.16:0/1701641973 wait complete. 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.713+0000 7f11298d3740 1 -- start start 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.713+0000 7f11298d3740 1 Processor -- start 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 0x5648717c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11208fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x564871841450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11208fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44506/0 (socket says 172.21.15.16:44506) 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1100011320 con 0x5648713e0980 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11208fa640 1 -- 172.21.15.16:0/3775375882 learned_addr learned my addr 172.21.15.16:0/3775375882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:35.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1100010c50 con 0x5648713e8b60 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11210fb640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 0x5648717c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.714+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f110000de90 con 0x5648713c81c0 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11218fc640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x564871841450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 -- 172.21.15.16:0/3775375882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 msgr2=0x5648717c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 0x5648717c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11298d3740 10 client.0 _ll_register_callbacks cb 0x7f110c0097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 -- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 msgr2=0x564871841450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x564871841450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648717c8b70 con 0x5648713e0980 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11210fb640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 0x5648717c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:08:35.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1118038070 con 0x5648713e0980 2024-09-21T02:08:35.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f11208fa640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f110c0ce3a0 tx=0x7f110c0d7ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4e470672fc4cd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:35.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.715+0000 7f1104ff9640 10 client.15792 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:08:35.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f1104ff9640 1 -- 172.21.15.16:0/3775375882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c0cdff0 con 0x5648713e0980 2024-09-21T02:08:35.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f1104ff9640 1 -- 172.21.15.16:0/3775375882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f110c0c9ff0 con 0x5648713e0980 2024-09-21T02:08:35.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f1104ff9640 1 -- 172.21.15.16:0/3775375882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f110c0ce470 con 0x5648713e0980 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 10 client.15792 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5648718428c0 con 0x5648713e0980 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f1104ff9640 1 -- 172.21.15.16:0/3775375882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f110c0f0160 con 0x5648713e0980 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 10 client.15792 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 10 client.15792 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 20 client.15792 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x564871409840 0x564871844e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 4 client.15792 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}] 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.716+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.0, "await_for": 45.0}) -- 0x564871724590 con 0x564871409840 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.717+0000 7f11210fb640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x564871409840 0x564871844e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.717+0000 7f11210fb640 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x564871409840 0x564871844e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93c606460c76055b server_cookie=ba42b82ffb8c9a1f in_seq=0 out_seq=0 2024-09-21T02:08:35.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.717+0000 7f1104ff9640 10 client.15792 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f1104ff9640 1 -- 172.21.15.16:0/3775375882 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x564871724590 con 0x564871409840 2024-09-21T02:08:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f1104ff9640 10 client.15792 handle_command_reply: tid=0 2024-09-21T02:08:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 client.15792 shutdown 2024-09-21T02:08:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 msgr2=0x5648717c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f110c0ce3a0 tx=0x7f110c0d7ee0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 shutdown_connections 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x564871409840 0x564871844e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5648713e8b60 0x564871841450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5648713e0980 0x5648717c82a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 --2- 172.21.15.16:0/3775375882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5648713c81c0 0x5648717c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f1104ff9640 0 client.15792 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 >> 172.21.15.16:0/3775375882 conn(0x5648713fd990 msgr2=0x5648717ba800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 shutdown_connections 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 1 -- 172.21.15.16:0/3775375882 wait complete. 2024-09-21T02:08:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.759+0000 7f11298d3740 20 client.15792 trim_cache size 0 max 16384 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 6, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "40a5f275": { 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 6, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:08:35.762 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 137.0, 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:35.763 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:08:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 msgr2=0x7f11080b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 0x7f11080b8ce0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5648717b7a90 tx=0x7f1118005f90 comp rx=0 tx=0).stop 2024-09-21T02:08:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 msgr2=0x7f111c19cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f110c000c00 tx=0x7f110c031630 comp rx=0 tx=0).stop 2024-09-21T02:08:35.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 shutdown_connections 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f11080b67d0 0x7f11080b8ce0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f111c110200 0x7f111c1a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f111c102fa0 0x7f111c19cb90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 --2- 172.21.15.16:0/25429960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f111c102650 0x7f111c19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.760+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 >> 172.21.15.16:0/25429960 conn(0x7f111c079760 msgr2=0x7f111c1116b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.761+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 shutdown_connections 2024-09-21T02:08:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:35.761+0000 7f112335f640 1 -- 172.21.15.16:0/25429960 wait complete. 2024-09-21T02:08:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:35 smithi016 ceph-mon[31245]: pgmap v395: 129 pgs: 129 active+clean; 884 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 74 MiB/s wr, 55 op/s 2024-09-21T02:08:36.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:35 smithi161 ceph-mon[34061]: pgmap v395: 129 pgs: 129 active+clean; 884 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 74 MiB/s wr, 55 op/s 2024-09-21T02:08:36.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:35 smithi123 ceph-mon[33453]: pgmap v395: 129 pgs: 129 active+clean; 884 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 74 MiB/s wr, 55 op/s 2024-09-21T02:08:36.573 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '40a5f275', quiesce took 4.4 seconds. Will release after: 12.6 2024-09-21T02:08:38.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:37 smithi016 ceph-mon[31245]: pgmap v396: 129 pgs: 129 active+clean; 1.0 GiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 88 MiB/s wr, 60 op/s 2024-09-21T02:08:38.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:37 smithi161 ceph-mon[34061]: pgmap v396: 129 pgs: 129 active+clean; 1.0 GiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 88 MiB/s wr, 60 op/s 2024-09-21T02:08:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:37 smithi123 ceph-mon[33453]: pgmap v396: 129 pgs: 129 active+clean; 1.0 GiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 88 MiB/s wr, 60 op/s 2024-09-21T02:08:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:39 smithi016 ceph-mon[31245]: pgmap v397: 129 pgs: 129 active+clean; 1.2 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 MiB/s wr, 66 op/s 2024-09-21T02:08:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:08:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:08:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:08:40.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:39 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:08:40.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:39 smithi161 ceph-mon[34061]: pgmap v397: 129 pgs: 129 active+clean; 1.2 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 MiB/s wr, 66 op/s 2024-09-21T02:08:40.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:08:40.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:08:40.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:08:40.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:39 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:08:40.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:39 smithi123 ceph-mon[33453]: pgmap v397: 129 pgs: 129 active+clean; 1.2 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 MiB/s wr, 66 op/s 2024-09-21T02:08:40.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:08:40.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:08:40.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:08:40.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:39 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:08:42.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:41 smithi016 ceph-mon[31245]: pgmap v398: 129 pgs: 129 active+clean; 1.3 GiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 108 MiB/s wr, 72 op/s 2024-09-21T02:08:42.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:41 smithi161 ceph-mon[34061]: pgmap v398: 129 pgs: 129 active+clean; 1.3 GiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 108 MiB/s wr, 72 op/s 2024-09-21T02:08:42.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:41 smithi123 ceph-mon[33453]: pgmap v398: 129 pgs: 129 active+clean; 1.3 GiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 108 MiB/s wr, 72 op/s 2024-09-21T02:08:44.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:43 smithi016 ceph-mon[31245]: pgmap v399: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 96 MiB/s wr, 62 op/s 2024-09-21T02:08:44.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:43 smithi161 ceph-mon[34061]: pgmap v399: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 96 MiB/s wr, 62 op/s 2024-09-21T02:08:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:43 smithi123 ceph-mon[33453]: pgmap v399: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 96 MiB/s wr, 62 op/s 2024-09-21T02:08:46.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:45 smithi016 ceph-mon[31245]: pgmap v400: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 74 MiB/s wr, 51 op/s 2024-09-21T02:08:46.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:45 smithi161 ceph-mon[34061]: pgmap v400: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 74 MiB/s wr, 51 op/s 2024-09-21T02:08:46.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:45 smithi123 ceph-mon[33453]: pgmap v400: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 74 MiB/s wr, 51 op/s 2024-09-21T02:08:48.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:47 smithi016 ceph-mon[31245]: pgmap v401: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-09-21T02:08:48.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:47 smithi161 ceph-mon[34061]: pgmap v401: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-09-21T02:08:48.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:47 smithi123 ceph-mon[33453]: pgmap v401: 129 pgs: 129 active+clean; 1.3 GiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-09-21T02:08:49.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:08:49.448 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:49.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 -- 172.21.15.16:0/3244727575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80102ae0 msgr2=0x7ffb80102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:49.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 --2- 172.21.15.16:0/3244727575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80102ae0 0x7ffb80102ef0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009900 tx=0x7ffb7002a3f0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 -- 172.21.15.16:0/3244727575 shutdown_connections 2024-09-21T02:08:49.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 --2- 172.21.15.16:0/3244727575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80108d20 0x7ffb8010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 --2- 172.21.15.16:0/3244727575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 0x7ffb801087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 --2- 172.21.15.16:0/3244727575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80102ae0 0x7ffb80102ef0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 -- 172.21.15.16:0/3244727575 >> 172.21.15.16:0/3244727575 conn(0x7ffb80079760 msgr2=0x7ffb80079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 -- 172.21.15.16:0/3244727575 shutdown_connections 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.970+0000 7ffb865b1640 1 -- 172.21.15.16:0/3244727575 wait complete. 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 Processor -- start 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 -- start start 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80102ae0 0x7ffb80199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 0x7ffb80199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb80112840 con 0x7ffb80108d20 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffb801126c0 con 0x7ffb80102ae0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb865b1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffb801129c0 con 0x7ffb801083d0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb84b4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb84b4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48968/0 (socket says 172.21.15.16:48968) 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb84b4e640 1 -- 172.21.15.16:0/449579799 learned_addr learned my addr 172.21.15.16:0/449579799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb77fff640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 0x7ffb80199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.971+0000 7ffb7ffff640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80102ae0 0x7ffb80199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 -- 172.21.15.16:0/449579799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 msgr2=0x7ffb80199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 0x7ffb80199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 -- 172.21.15.16:0/449579799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80102ae0 msgr2=0x7ffb80199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80102ae0 0x7ffb80199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 -- 172.21.15.16:0/449579799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8019f140 con 0x7ffb80108d20 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb84b4e640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800cce0 tx=0x7ffb7800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7800f450 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb7800fa80 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8019f430 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78018ad0 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.972+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8006bb40 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.973+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb80103d10 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.975+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ffb7800f5f0 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.976+0000 7ffb7dffb640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 0x7ffb6c0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.976+0000 7ffb7ffff640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 0x7ffb6c0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.976+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ffb780e8290 con 0x7ffb80108d20 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.977+0000 7ffb7ffff640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 0x7ffb6c0b8bc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb700057c0 tx=0x7ffb700058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:49.980+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ffb78014030 con 0x7ffb80108d20 2024-09-21T02:08:50.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:49 smithi016 ceph-mon[31245]: pgmap v402: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-09-21T02:08:50.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:49 smithi161 ceph-mon[34061]: pgmap v402: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-09-21T02:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.205+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb80102ef0 con 0x7ffb80108d20 2024-09-21T02:08:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.206+0000 7ffb7dffb640 1 -- 172.21.15.16:0/449579799 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7ffb80102ef0 con 0x7ffb80108d20 2024-09-21T02:08:50.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:08:50.207 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:08:50.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 msgr2=0x7ffb6c0b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:50.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 0x7ffb6c0b8bc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb700057c0 tx=0x7ffb700058d0 comp rx=0 tx=0).stop 2024-09-21T02:08:50.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 msgr2=0x7ffb8019ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800cce0 tx=0x7ffb7800a620 comp rx=0 tx=0).stop 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 shutdown_connections 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ffb6c0b66b0 0x7ffb6c0b8bc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb80108d20 0x7ffb8019ead0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb801083d0 0x7ffb80199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 --2- 172.21.15.16:0/449579799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb80102ae0 0x7ffb80199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.208+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 >> 172.21.15.16:0/449579799 conn(0x7ffb80079760 msgr2=0x7ffb80101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.209+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 shutdown_connections 2024-09-21T02:08:50.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:50.209+0000 7ffb865b1640 1 -- 172.21.15.16:0/449579799 wait complete. 2024-09-21T02:08:50.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:49 smithi123 ceph-mon[33453]: pgmap v402: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-09-21T02:08:51.023 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 40a5f275 --release --await' 2024-09-21T02:08:51.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 40a5f275 --release --await 2024-09-21T02:08:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:50 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/449579799' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:51.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:50 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/449579799' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:51.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:50 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/449579799' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:08:51.284 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.794+0000 7f19624a9640 1 -- 172.21.15.16:0/2633372143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c102ae0 msgr2=0x7f195c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.794+0000 7f19624a9640 1 --2- 172.21.15.16:0/2633372143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c102ae0 0x7f195c102ef0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f194c009900 tx=0x7f194c02a410 comp rx=0 tx=0).stop 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 -- 172.21.15.16:0/2633372143 shutdown_connections 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 --2- 172.21.15.16:0/2633372143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c108d20 0x7f195c10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 --2- 172.21.15.16:0/2633372143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 --2- 172.21.15.16:0/2633372143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c102ae0 0x7f195c102ef0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 -- 172.21.15.16:0/2633372143 >> 172.21.15.16:0/2633372143 conn(0x7f195c079760 msgr2=0x7f195c079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 -- 172.21.15.16:0/2633372143 shutdown_connections 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 -- 172.21.15.16:0/2633372143 wait complete. 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 Processor -- start 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.795+0000 7f19624a9640 1 -- start start 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 0x7f195c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112840 con 0x7f195c108d20 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1126c0 con 0x7f195c1083d0 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f19624a9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1129c0 con 0x7f195c102ae0 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f195bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 0x7f195c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48978/0 (socket says 172.21.15.16:48978) 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 -- 172.21.15.16:0/2179453014 learned_addr learned my addr 172.21.15.16:0/2179453014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f195bfff640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 0x7f195c199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:46902/0 (socket says 172.21.15.16:46902) 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f195b7fe640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 -- 172.21.15.16:0/2179453014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 msgr2=0x7f195c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 0x7f195c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 -- 172.21.15.16:0/2179453014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 msgr2=0x7f195c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f1960a46640 1 -- 172.21.15.16:0/2179453014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c19f200 con 0x7f195c108d20 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.796+0000 7f195b7fe640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f1960a46640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f195400ebb0 tx=0x7f1954007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f19597fa640 1 -- 172.21.15.16:0/2179453014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954007ca0 con 0x7f195c108d20 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c19f490 con 0x7f195c108d20 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f19597fa640 1 -- 172.21.15.16:0/2179453014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1954007e40 con 0x7f195c108d20 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f19597fa640 1 -- 172.21.15.16:0/2179453014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954005760 con 0x7f195c108d20 2024-09-21T02:08:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.797+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c06bb40 con 0x7f195c108d20 2024-09-21T02:08:51.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f19597fa640 1 -- 172.21.15.16:0/2179453014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f1954004a40 con 0x7f195c108d20 2024-09-21T02:08:51.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 Processor -- start 2024-09-21T02:08:51.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 -- start start 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987c8660 0x561098b97a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098b97f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561098a82dc0 con 0x5610987f1520 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561098b0f1f0 con 0x5610987c8660 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f1968a1d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561098afe620 con 0x5610987d0840 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48984/0 (socket says 172.21.15.16:48984) 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f195b7fe640 1 -- 172.21.15.16:0/1485381293 learned_addr learned my addr 172.21.15.16:0/1485381293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f195b7fe640 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098b97f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.800+0000 7f195bfff640 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987c8660 0x561098b97a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f19597fa640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 0x7f19440b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 -- 172.21.15.16:0/1485381293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 msgr2=0x561098b97f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098b97f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 -- 172.21.15.16:0/1485381293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987c8660 msgr2=0x561098b97a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987c8660 0x561098b97a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 -- 172.21.15.16:0/1485381293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561098ac6fd0 con 0x5610987f1520 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f1960a46640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 0x7f19440b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f195b7fe640 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1950013760 tx=0x7f1950009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e56719b6120c067 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f19597fa640 1 -- 172.21.15.16:0/2179453014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f19540e9090 con 0x7f195c108d20 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f19427fc640 1 -- 172.21.15.16:0/1485381293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950013080 con 0x5610987f1520 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f19427fc640 1 -- 172.21.15.16:0/1485381293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19500044c0 con 0x5610987f1520 2024-09-21T02:08:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.801+0000 7f19427fc640 1 -- 172.21.15.16:0/1485381293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000fdf0 con 0x5610987f1520 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 -- 172.21.15.16:0/1485381293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 msgr2=0x561098b97510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1950013760 tx=0x7f1950009790 comp rx=0 tx=0).stop 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1960a46640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 0x7f19440b8b90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f19540ae870 tx=0x7f1954073100 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 -- 172.21.15.16:0/1485381293 shutdown_connections 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098b97f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987c8660 0x561098b97a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 --2- 172.21.15.16:0/1485381293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987f1520 0x561098b97510 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 -- 172.21.15.16:0/1485381293 >> 172.21.15.16:0/1485381293 conn(0x5610987e5910 msgr2=0x561098b95400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 -- 172.21.15.16:0/1485381293 shutdown_connections 2024-09-21T02:08:51.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.802+0000 7f1968a1d740 1 -- 172.21.15.16:0/1485381293 wait complete. 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 -- start start 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 Processor -- start 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 0x561098ba3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987f1520 0x561098bb3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561098a99300 con 0x5610987c8660 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561098b02430 con 0x5610987d0840 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56109895fbb0 con 0x5610987f1520 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f1968a1d740 10 client.0 _ll_register_callbacks cb 0x561098ab3120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48986/0 (socket says 172.21.15.16:48986) 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.803+0000 7f195b7fe640 1 -- 172.21.15.16:0/4131086028 learned_addr learned my addr 172.21.15.16:0/4131086028 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195bfff640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 0x561098ba3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 -- 172.21.15.16:0/4131086028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987f1520 msgr2=0x561098bb3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987f1520 0x561098bb3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 -- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 msgr2=0x561098ba3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 0x561098ba3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 -- 172.21.15.16:0/4131086028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561098a88780 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195bfff640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 0x561098ba3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f195b7fe640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d20 tx=0x7f19500058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4520fcd75cec68a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f193b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950005220 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f19500053c0 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561098b11140 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f195002f050 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f1968a1d740 10 client.15810 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.804+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56109885cb00 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x561098b11140 con 0x5610987c8660 2024-09-21T02:08:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f1950020560 con 0x5610987c8660 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 10 client.15810 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 10 client.15810 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 20 client.15810 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987afea0 0x561098bb91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 4 client.15810 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561098b0cbf0 con 0x5610987afea0 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.805+0000 7f195bfff640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987afea0 0x561098bb91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.806+0000 7f193b7fe640 10 client.15810 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.806+0000 7f195bfff640 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987afea0 0x561098bb91e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4119ba38fbef10 server_cookie=303a652352a31b88 in_seq=0 out_seq=0 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.806+0000 7f193b7fe640 1 -- 172.21.15.16:0/4131086028 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561098b0cbf0 con 0x5610987afea0 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.806+0000 7f193b7fe640 10 client.15810 handle_command_reply: tid=0 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.806+0000 7f1968a1d740 1 client.15810 shutdown 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 msgr2=0x561098ba2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d20 tx=0x7f19500058b0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 shutdown_connections 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987afea0 0x561098bb91e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f193b7fe640 0 client.15810 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987f1520 0x561098bb3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987d0840 0x561098ba3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 --2- 172.21.15.16:0/4131086028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098ba2c00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 >> 172.21.15.16:0/4131086028 conn(0x5610987e5910 msgr2=0x561098b9a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 shutdown_connections 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 1 -- 172.21.15.16:0/4131086028 wait complete. 2024-09-21T02:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.807+0000 7f1968a1d740 20 client.15810 trim_cache size 0 max 16384 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.824+0000 7f1968a1d740 1 Processor -- start 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 -- start start 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098ba4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987c8660 0x561098b99020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1aeb50 con 0x5610987d0840 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1b0690 con 0x5610987afea0 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f1968a1d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1af220 con 0x5610987c8660 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48996/0 (socket says 172.21.15.16:48996) 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 -- 172.21.15.16:0/241924559 learned_addr learned my addr 172.21.15.16:0/241924559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098ba4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195bfff640 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987c8660 0x561098b99020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 -- 172.21.15.16:0/241924559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987c8660 msgr2=0x561098b99020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987c8660 0x561098b99020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 -- 172.21.15.16:0/241924559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 msgr2=0x561098ba4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098ba4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.825+0000 7f195b7fe640 1 -- 172.21.15.16:0/241924559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610988d3960 con 0x5610987d0840 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.826+0000 7f195b7fe640 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f195000f830 tx=0x7f1950009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfe83d1df075291 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.826+0000 7f1941ffb640 1 -- 172.21.15.16:0/241924559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950020af0 con 0x5610987d0840 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.826+0000 7f1941ffb640 1 -- 172.21.15.16:0/241924559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1950020c90 con 0x5610987d0840 2024-09-21T02:08:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.826+0000 7f1941ffb640 1 -- 172.21.15.16:0/241924559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950020710 con 0x5610987d0840 2024-09-21T02:08:51.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- 172.21.15.16:0/241924559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 msgr2=0x561098b9c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f195000f830 tx=0x7f1950009f00 comp rx=0 tx=0).stop 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- 172.21.15.16:0/241924559 shutdown_connections 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987d0840 0x561098b9c790 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987c8660 0x561098b99020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 --2- 172.21.15.16:0/241924559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098ba4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- 172.21.15.16:0/241924559 >> 172.21.15.16:0/241924559 conn(0x5610987e5910 msgr2=0x561098ba1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- 172.21.15.16:0/241924559 shutdown_connections 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- 172.21.15.16:0/241924559 wait complete. 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 -- start start 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.827+0000 7f1968a1d740 1 Processor -- start 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098bafaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098bab640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1affc0 con 0x5610987c8660 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1af8f0 con 0x5610987afea0 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1acb30 con 0x5610987d0840 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f1968a1d740 10 client.0 _ll_register_callbacks cb 0x7f1950009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49000/0 (socket says 172.21.15.16:49000) 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 -- 172.21.15.16:0/3948955462 learned_addr learned my addr 172.21.15.16:0/3948955462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195b7fe640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098bab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195b7fe640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098bafaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 -- 172.21.15.16:0/3948955462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 msgr2=0x561098bab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098bab640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 -- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 msgr2=0x561098bafaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098bafaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 -- 172.21.15.16:0/3948955462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561098babdb0 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f195bfff640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f194c0057c0 tx=0x7f194c031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7af973bd32d873 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.828+0000 7f193b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c035040 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f194c004240 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950020c90 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f194c055050 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 10 client.15816 fetch_fsmap learned FSMap version 28 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561098c23af0 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1950020c90 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f194c03b430 con 0x5610987c8660 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 10 client.15816 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 10 client.15816 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 20 client.15816 populate_metadata read hostname 'smithi016' 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987f1520 0x561098c26290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 4 client.15816 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "40a5f275", "release": true, "await": true}] 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.829+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "40a5f275", "release": true, "await": true}) -- 0x561098b0a020 con 0x5610987f1520 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.830+0000 7f195b7fe640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987f1520 0x561098c26290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.830+0000 7f193b7fe640 10 client.15816 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:51.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.830+0000 7f195b7fe640 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987f1520 0x561098c26290 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63e9c030feec4fe5 server_cookie=23a2b5428fca1493 in_seq=0 out_seq=0 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.835+0000 7f193b7fe640 1 -- 172.21.15.16:0/3948955462 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x561098b0a020 con 0x5610987f1520 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.835+0000 7f193b7fe640 10 client.15816 handle_command_reply: tid=0 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.835+0000 7f1968a1d740 1 client.15816 shutdown 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 msgr2=0x561098baaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f194c0057c0 tx=0x7f194c031740 comp rx=0 tx=0).stop 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 shutdown_connections 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5610987f1520 0x561098c26290 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f193b7fe640 0 client.15816 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:08:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5610987d0840 0x561098bab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5610987c8660 0x561098baaff0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 --2- 172.21.15.16:0/3948955462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5610987afea0 0x561098bafaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 >> 172.21.15.16:0/3948955462 conn(0x5610987e5910 msgr2=0x561098ba1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 shutdown_connections 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 1 -- 172.21.15.16:0/3948955462 wait complete. 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.836+0000 7f1968a1d740 20 client.15816 trim_cache size 0 max 16384 2024-09-21T02:08:51.838 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 8, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "40a5f275": { 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 8, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 137.0, 2024-09-21T02:08:51.839 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:08:51.840 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 msgr2=0x7f19440b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 0x7f19440b8b90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f19540ae870 tx=0x7f1954073100 comp rx=0 tx=0).stop 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 msgr2=0x7f195c19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f195400ebb0 tx=0x7f1954007590 comp rx=0 tx=0).stop 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 shutdown_connections 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19440b6680 0x7f19440b8b90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f195c108d20 0x7f195c19ead0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f195c1083d0 0x7f195c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 --2- 172.21.15.16:0/2179453014 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f195c102ae0 0x7f195c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:08:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 >> 172.21.15.16:0/2179453014 conn(0x7f195c079760 msgr2=0x7f195c1014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:08:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.837+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 shutdown_connections 2024-09-21T02:08:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:08:51.838+0000 7f19624a9640 1 -- 172.21.15.16:0/2179453014 wait complete. 2024-09-21T02:08:52.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:51 smithi016 ceph-mon[31245]: pgmap v403: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 17 op/s 2024-09-21T02:08:52.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:51 smithi161 ceph-mon[34061]: pgmap v403: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 17 op/s 2024-09-21T02:08:52.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:51 smithi123 ceph-mon[33453]: pgmap v403: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 17 op/s 2024-09-21T02:08:52.781 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '40a5f275', total seconds elapsed: 20.6 2024-09-21T02:08:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:52 smithi016 ceph-mon[31245]: pgmap v404: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 673 KiB/s wr, 10 op/s 2024-09-21T02:08:53.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:52 smithi161 ceph-mon[34061]: pgmap v404: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 673 KiB/s wr, 10 op/s 2024-09-21T02:08:53.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:52 smithi123 ceph-mon[33453]: pgmap v404: 129 pgs: 129 active+clean; 1.3 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 673 KiB/s wr, 10 op/s 2024-09-21T02:08:55.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:55 smithi016 ceph-mon[31245]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 15 MiB/s wr, 19 op/s 2024-09-21T02:08:55.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:55 smithi161 ceph-mon[34061]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 15 MiB/s wr, 19 op/s 2024-09-21T02:08:55.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:55 smithi123 ceph-mon[33453]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 15 MiB/s wr, 19 op/s 2024-09-21T02:08:57.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:57 smithi016 ceph-mon[31245]: pgmap v406: 129 pgs: 129 active+clean; 1.6 GiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-09-21T02:08:57.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:57 smithi161 ceph-mon[34061]: pgmap v406: 129 pgs: 129 active+clean; 1.6 GiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-09-21T02:08:57.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:57 smithi123 ceph-mon[33453]: pgmap v406: 129 pgs: 129 active+clean; 1.6 GiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-09-21T02:08:59.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:08:59 smithi016 ceph-mon[31245]: pgmap v407: 129 pgs: 129 active+clean; 1.8 GiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:08:59.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:08:59 smithi161 ceph-mon[34061]: pgmap v407: 129 pgs: 129 active+clean; 1.8 GiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:08:59.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:08:59 smithi123 ceph-mon[33453]: pgmap v407: 129 pgs: 129 active+clean; 1.8 GiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:09:01.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:01 smithi016 ceph-mon[31245]: pgmap v408: 129 pgs: 129 active+clean; 2.2 GiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 73 MiB/s wr, 48 op/s 2024-09-21T02:09:01.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:01 smithi161 ceph-mon[34061]: pgmap v408: 129 pgs: 129 active+clean; 2.2 GiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 73 MiB/s wr, 48 op/s 2024-09-21T02:09:01.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:01 smithi123 ceph-mon[33453]: pgmap v408: 129 pgs: 129 active+clean; 2.2 GiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 73 MiB/s wr, 48 op/s 2024-09-21T02:09:03.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:03 smithi161 ceph-mon[34061]: pgmap v409: 129 pgs: 129 active+clean; 2.4 GiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 59 op/s 2024-09-21T02:09:03.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:03 smithi123 ceph-mon[33453]: pgmap v409: 129 pgs: 129 active+clean; 2.4 GiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 59 op/s 2024-09-21T02:09:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:03 smithi016 ceph-mon[31245]: pgmap v409: 129 pgs: 129 active+clean; 2.4 GiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 98 MiB/s wr, 59 op/s 2024-09-21T02:09:05.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:05 smithi161 ceph-mon[34061]: pgmap v410: 129 pgs: 129 active+clean; 2.9 GiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 139 MiB/s wr, 97 op/s 2024-09-21T02:09:05.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:05 smithi123 ceph-mon[33453]: pgmap v410: 129 pgs: 129 active+clean; 2.9 GiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 139 MiB/s wr, 97 op/s 2024-09-21T02:09:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:05 smithi016 ceph-mon[31245]: pgmap v410: 129 pgs: 129 active+clean; 2.9 GiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 139 MiB/s wr, 97 op/s 2024-09-21T02:09:07.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:07 smithi161 ceph-mon[34061]: pgmap v411: 129 pgs: 129 active+clean; 3.1 GiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 141 MiB/s wr, 96 op/s 2024-09-21T02:09:07.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:07 smithi123 ceph-mon[33453]: pgmap v411: 129 pgs: 129 active+clean; 3.1 GiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 141 MiB/s wr, 96 op/s 2024-09-21T02:09:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:07 smithi016 ceph-mon[31245]: pgmap v411: 129 pgs: 129 active+clean; 3.1 GiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 141 MiB/s wr, 96 op/s 2024-09-21T02:09:09.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:09 smithi123 ceph-mon[33453]: pgmap v412: 129 pgs: 129 active+clean; 3.3 GiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 148 MiB/s wr, 99 op/s 2024-09-21T02:09:09.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:09 smithi016 ceph-mon[31245]: pgmap v412: 129 pgs: 129 active+clean; 3.3 GiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 148 MiB/s wr, 99 op/s 2024-09-21T02:09:09.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:09 smithi161 ceph-mon[34061]: pgmap v412: 129 pgs: 129 active+clean; 3.3 GiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 148 MiB/s wr, 99 op/s 2024-09-21T02:09:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:11 smithi123 ceph-mon[33453]: pgmap v413: 129 pgs: 129 active+clean; 3.7 GiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 163 MiB/s wr, 108 op/s 2024-09-21T02:09:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:11 smithi016 ceph-mon[31245]: pgmap v413: 129 pgs: 129 active+clean; 3.7 GiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 163 MiB/s wr, 108 op/s 2024-09-21T02:09:11.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:11 smithi161 ceph-mon[34061]: pgmap v413: 129 pgs: 129 active+clean; 3.7 GiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 163 MiB/s wr, 108 op/s 2024-09-21T02:09:13.460 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:13 smithi016 ceph-mon[31245]: pgmap v414: 129 pgs: 129 active+clean; 4.0 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 156 MiB/s wr, 103 op/s 2024-09-21T02:09:13.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:13 smithi123 ceph-mon[33453]: pgmap v414: 129 pgs: 129 active+clean; 4.0 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 156 MiB/s wr, 103 op/s 2024-09-21T02:09:13.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:13 smithi161 ceph-mon[34061]: pgmap v414: 129 pgs: 129 active+clean; 4.0 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 156 MiB/s wr, 103 op/s 2024-09-21T02:09:15.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:15 smithi123 ceph-mon[33453]: pgmap v415: 129 pgs: 129 active+clean; 4.4 GiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 170 MiB/s wr, 113 op/s 2024-09-21T02:09:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:15 smithi016 ceph-mon[31245]: pgmap v415: 129 pgs: 129 active+clean; 4.4 GiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 170 MiB/s wr, 113 op/s 2024-09-21T02:09:15.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:15 smithi161 ceph-mon[34061]: pgmap v415: 129 pgs: 129 active+clean; 4.4 GiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 170 MiB/s wr, 113 op/s 2024-09-21T02:09:17.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:17 smithi123 ceph-mon[33453]: pgmap v416: 129 pgs: 129 active+clean; 4.6 GiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 144 MiB/s wr, 82 op/s 2024-09-21T02:09:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:17 smithi016 ceph-mon[31245]: pgmap v416: 129 pgs: 129 active+clean; 4.6 GiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 144 MiB/s wr, 82 op/s 2024-09-21T02:09:17.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:17 smithi161 ceph-mon[34061]: pgmap v416: 129 pgs: 129 active+clean; 4.6 GiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 144 MiB/s wr, 82 op/s 2024-09-21T02:09:19.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:19 smithi123 ceph-mon[33453]: pgmap v417: 129 pgs: 129 active+clean; 4.8 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 147 MiB/s wr, 83 op/s 2024-09-21T02:09:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:19 smithi016 ceph-mon[31245]: pgmap v417: 129 pgs: 129 active+clean; 4.8 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 147 MiB/s wr, 83 op/s 2024-09-21T02:09:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:19 smithi161 ceph-mon[34061]: pgmap v417: 129 pgs: 129 active+clean; 4.8 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 147 MiB/s wr, 83 op/s 2024-09-21T02:09:21.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:21 smithi123 ceph-mon[33453]: pgmap v418: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 157 MiB/s wr, 90 op/s 2024-09-21T02:09:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:21 smithi016 ceph-mon[31245]: pgmap v418: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 157 MiB/s wr, 90 op/s 2024-09-21T02:09:21.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:21 smithi161 ceph-mon[34061]: pgmap v418: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 157 MiB/s wr, 90 op/s 2024-09-21T02:09:23.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:23 smithi123 ceph-mon[33453]: pgmap v419: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 156 MiB/s wr, 88 op/s 2024-09-21T02:09:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:23 smithi016 ceph-mon[31245]: pgmap v419: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 156 MiB/s wr, 88 op/s 2024-09-21T02:09:23.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:23 smithi161 ceph-mon[34061]: pgmap v419: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 156 MiB/s wr, 88 op/s 2024-09-21T02:09:24.353 INFO:tasks.quiescer.fs.[cephfs]:Total duration (55.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:09:24.353 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T02:09:24.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:09:24.733 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 -- 172.21.15.16:0/2996901457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 msgr2=0x7fefa41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/2996901457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa41098c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009900 tx=0x7fef8c02a410 comp rx=0 tx=0).stop 2024-09-21T02:09:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 -- 172.21.15.16:0/2996901457 shutdown_connections 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/2996901457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa4110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/2996901457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa41098c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/2996901457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 -- 172.21.15.16:0/2996901457 >> 172.21.15.16:0/2996901457 conn(0x7fefa40ff5e0 msgr2=0x7fefa4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 -- 172.21.15.16:0/2996901457 shutdown_connections 2024-09-21T02:09:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.382+0000 7fefa8a7f640 1 -- 172.21.15.16:0/2996901457 wait complete. 2024-09-21T02:09:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.385+0000 7fefa8a7f640 1 Processor -- start 2024-09-21T02:09:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.385+0000 7fefa8a7f640 1 -- start start 2024-09-21T02:09:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa41126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fefa4077100 con 0x7fefa41094b0 2024-09-21T02:09:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fefa4076f80 con 0x7fefa4109e00 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa8a7f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fefa4077280 con 0x7fefa4103b70 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47946/0 (socket says 172.21.15.16:47946) 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fef9bfff640 1 -- 172.21.15.16:0/4205646879 learned_addr learned my addr 172.21.15.16:0/4205646879 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa2d9d640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa41126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.386+0000 7fefa259c640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 -- 172.21.15.16:0/4205646879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 msgr2=0x7fefa4115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 -- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 msgr2=0x7fefa4111f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 -- 172.21.15.16:0/4205646879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa41acb70 con 0x7fefa4109e00 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa259c640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fef9bfff640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa2d9d640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa41126a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00da10 tx=0x7fef9c00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00bd70 con 0x7fefa4109e00 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa41acda0 con 0x7fefa4109e00 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c003e00 con 0x7fefa4109e00 2024-09-21T02:09:25.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0106c0 con 0x7fefa4109e00 2024-09-21T02:09:25.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.387+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa41ad1f0 con 0x7fefa4109e00 2024-09-21T02:09:25.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.390+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa4103f80 con 0x7fefa4109e00 2024-09-21T02:09:25.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.391+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fef9c003970 con 0x7fefa4109e00 2024-09-21T02:09:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.392+0000 7fef9b7fe640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 0x7fef900b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.392+0000 7fefa259c640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 0x7fef900b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:25.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.393+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fef9c0e7ed0 con 0x7fefa4109e00 2024-09-21T02:09:25.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.393+0000 7fefa259c640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 0x7fef900b8c10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefa41159f0 tx=0x7fef94005e70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:25.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.397+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fef9c0ae0b0 con 0x7fefa4109e00 2024-09-21T02:09:25.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:25 smithi123 ceph-mon[33453]: pgmap v420: 129 pgs: 129 active+clean; 6.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 168 MiB/s wr, 96 op/s 2024-09-21T02:09:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:25 smithi016 ceph-mon[31245]: pgmap v420: 129 pgs: 129 active+clean; 6.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 168 MiB/s wr, 96 op/s 2024-09-21T02:09:25.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:25 smithi161 ceph-mon[34061]: pgmap v420: 129 pgs: 129 active+clean; 6.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 168 MiB/s wr, 96 op/s 2024-09-21T02:09:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.795+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefa402e640 con 0x7fefa4109e00 2024-09-21T02:09:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.796+0000 7fef9b7fe640 1 -- 172.21.15.16:0/4205646879 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fef9c0b2490 con 0x7fefa4109e00 2024-09-21T02:09:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:09:25.797 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:09:25.797 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 msgr2=0x7fef900b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 0x7fef900b8c10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefa41159f0 tx=0x7fef94005e70 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 msgr2=0x7fefa41126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa41126a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00da10 tx=0x7fef9c00ded0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 shutdown_connections 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fef900b6700 0x7fef900b8c10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fefa4109e00 0x7fefa41126a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa41094b0 0x7fefa4111f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 --2- 172.21.15.16:0/4205646879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefa4103b70 0x7fefa4115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.800+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 >> 172.21.15.16:0/4205646879 conn(0x7fefa40ff5e0 msgr2=0x7fefa4101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.801+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 shutdown_connections 2024-09-21T02:09:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:25.801+0000 7fefa8a7f640 1 -- 172.21.15.16:0/4205646879 wait complete. 2024-09-21T02:09:26.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4205646879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:26.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4205646879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:26.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4205646879' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:26.673 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T02:09:26.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T02:09:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:27 smithi123 ceph-mon[33453]: pgmap v421: 129 pgs: 129 active+clean; 6.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 144 MiB/s wr, 81 op/s 2024-09-21T02:09:27.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:27 smithi016 ceph-mon[31245]: pgmap v421: 129 pgs: 129 active+clean; 6.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 144 MiB/s wr, 81 op/s 2024-09-21T02:09:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:27 smithi161 ceph-mon[34061]: pgmap v421: 129 pgs: 129 active+clean; 6.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 144 MiB/s wr, 81 op/s 2024-09-21T02:09:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 -- 172.21.15.16:0/3171506330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 msgr2=0x7fdcbc102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 --2- 172.21.15.16:0/3171506330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc102e90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009900 tx=0x7fdcac02a410 comp rx=0 tx=0).stop 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 -- 172.21.15.16:0/3171506330 shutdown_connections 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 --2- 172.21.15.16:0/3171506330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc10f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 --2- 172.21.15.16:0/3171506330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 --2- 172.21.15.16:0/3171506330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc102e90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 -- 172.21.15.16:0/3171506330 >> 172.21.15.16:0/3171506330 conn(0x7fdcbc079760 msgr2=0x7fdcbc079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 -- 172.21.15.16:0/3171506330 shutdown_connections 2024-09-21T02:09:27.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.691+0000 7fdcc130a640 1 -- 172.21.15.16:0/3171506330 wait complete. 2024-09-21T02:09:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.692+0000 7fdcc130a640 1 Processor -- start 2024-09-21T02:09:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.692+0000 7fdcc130a640 1 -- start start 2024-09-21T02:09:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc1117e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc110310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc113940 con 0x7fdcbc102a80 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1137c0 con 0x7fdcbc1083c0 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcc130a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc113ac0 con 0x7fdcbc108d10 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:39838/0 (socket says 172.21.15.16:39838) 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 -- 172.21.15.16:0/121164053 learned_addr learned my addr 172.21.15.16:0/121164053 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc110310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc1117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 -- 172.21.15.16:0/121164053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 msgr2=0x7fdcbc110310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc110310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 -- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 msgr2=0x7fdcbc1117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.693+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc1117e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcb3fff640 1 -- 172.21.15.16:0/121164053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc110b30 con 0x7fdcbc1083c0 2024-09-21T02:09:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:09:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdca400e9c0 tx=0x7fdca400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcb8ff9640 1 -- 172.21.15.16:0/121164053 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca400cd80 con 0x7fdcbc1083c0 2024-09-21T02:09:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcb8ff9640 1 -- 172.21.15.16:0/121164053 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdca40054b0 con 0x7fdcbc1083c0 2024-09-21T02:09:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc1acd30 con 0x7fdcbc1083c0 2024-09-21T02:09:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcb8ff9640 1 -- 172.21.15.16:0/121164053 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4010430 con 0x7fdcbc1083c0 2024-09-21T02:09:27.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.694+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc1ad200 con 0x7fdcbc1083c0 2024-09-21T02:09:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.698+0000 7fdcb8ff9640 1 -- 172.21.15.16:0/121164053 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fdca4005020 con 0x7fdcbc1083c0 2024-09-21T02:09:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.698+0000 7fdcc787e740 1 Processor -- start 2024-09-21T02:09:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.699+0000 7fdcc787e740 1 -- start start 2024-09-21T02:09:27.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.699+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.699+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1ba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcc787e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bfbd12fb90 con 0x55bfbce12840 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcc787e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55bfbd11efc0 con 0x55bfbcdf1b60 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcc787e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55bfbd122dd0 con 0x55bfbcde9980 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcbb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59102/0 (socket says 172.21.15.16:59102) 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcbb7fe640 1 -- 172.21.15.16:0/2040744173 learned_addr learned my addr 172.21.15.16:0/2040744173 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:27.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.700+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcbb7fe640 1 -- 172.21.15.16:0/2040744173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 msgr2=0x55bfbd1b9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1b9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcbb7fe640 1 -- 172.21.15.16:0/2040744173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 msgr2=0x55bfbd1ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1ba080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcbb7fe640 1 -- 172.21.15.16:0/2040744173 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfbd0a6060 con 0x55bfbce12840 2024-09-21T02:09:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.701+0000 7fdcb8ff9640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 0x7fdca80b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.702+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 0x7fdca80b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.702+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4010480 tx=0x7fdcb4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a007a3c42c903f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.702+0000 7fdcb17fa640 1 -- 172.21.15.16:0/2040744173 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4011760 con 0x55bfbce12840 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.702+0000 7fdcb17fa640 1 -- 172.21.15.16:0/2040744173 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdcb4011d90 con 0x55bfbce12840 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.702+0000 7fdcb17fa640 1 -- 172.21.15.16:0/2040744173 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb401add0 con 0x55bfbce12840 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.703+0000 7fdcb8ff9640 1 -- 172.21.15.16:0/121164053 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fdca40e80e0 con 0x7fdcbc1083c0 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 0x7fdca80b8c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 -- 172.21.15.16:0/2040744173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 msgr2=0x55bfbd1b9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4010480 tx=0x7fdcb4009f90 comp rx=0 tx=0).stop 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 -- 172.21.15.16:0/2040744173 shutdown_connections 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1b9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2040744173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1b9600 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 -- 172.21.15.16:0/2040744173 >> 172.21.15.16:0/2040744173 conn(0x55bfbce06990 msgr2=0x55bfbd1b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 -- 172.21.15.16:0/2040744173 shutdown_connections 2024-09-21T02:09:27.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.704+0000 7fdcc787e740 1 -- 172.21.15.16:0/2040744173 wait complete. 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.705+0000 7fdcc787e740 1 -- start start 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.705+0000 7fdcc787e740 1 Processor -- start 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.705+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.705+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.705+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbce12840 0x55bfbd1d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcc787e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bfbcf807a0 con 0x55bfbcde9980 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcc787e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55bfbd0c40a0 con 0x55bfbce12840 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcc787e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55bfbd1c7de0 con 0x55bfbcdf1b60 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcc787e740 10 client.0 _ll_register_callbacks cb 0x55bfbd0a13c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59118/0 (socket says 172.21.15.16:59118) 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcbaffd640 1 -- 172.21.15.16:0/1768817207 learned_addr learned my addr 172.21.15.16:0/1768817207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbce12840 0x55bfbd1d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 -- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbce12840 msgr2=0x55bfbd1d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbce12840 0x55bfbd1d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 -- 172.21.15.16:0/1768817207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 msgr2=0x55bfbd1c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcb3fff640 1 -- 172.21.15.16:0/1768817207 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfbd0b2880 con 0x55bfbcdf1b60 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.706+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c5740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdca40e54c0 tx=0x7fdca40aa210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcc0a78ba570e7de server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdca27fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40ad990 con 0x55bfbcdf1b60 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcb4011d90 con 0x55bfbcdf1b60 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fdca40ea110 con 0x55bfbcdf1b60 2024-09-21T02:09:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdca4005310 con 0x55bfbcdf1b60 2024-09-21T02:09:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdcc787e740 10 client.25525 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bfbce8cb90 con 0x55bfbcdf1b60 2024-09-21T02:09:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.707+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdcb4011d90 con 0x55bfbcdf1b60 2024-09-21T02:09:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fdca40ace20 con 0x55bfbcdf1b60 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 10 client.25525 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 10 client.25525 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 20 client.25525 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcdd11c0 0x55bfbd1d7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 4 client.25525 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfbd131ae0 con 0x55bfbcdd11c0 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.708+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcdd11c0 0x55bfbd1d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.709+0000 7fdca27fc640 10 client.25525 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.709+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcdd11c0 0x55bfbd1d7130 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d05296cc7b0da2ca server_cookie=c4e2aae266fa65c5 in_seq=0 out_seq=0 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.709+0000 7fdca27fc640 1 -- 172.21.15.16:0/1768817207 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55bfbd131ae0 con 0x55bfbcdd11c0 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.709+0000 7fdca27fc640 10 client.25525 handle_command_reply: tid=0 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.709+0000 7fdcc787e740 1 client.25525 shutdown 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 msgr2=0x55bfbd1c5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c5740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdca40e54c0 tx=0x7fdca40aa210 comp rx=0 tx=0).stop 2024-09-21T02:09:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 shutdown_connections 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcdd11c0 0x55bfbd1d7130 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbce12840 0x55bfbd1d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c5740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1768817207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcde9980 0x55bfbd1c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 >> 172.21.15.16:0/1768817207 conn(0x55bfbce06990 msgr2=0x55bfbd1bc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdca27fc640 0 client.25525 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 shutdown_connections 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 1 -- 172.21.15.16:0/1768817207 wait complete. 2024-09-21T02:09:27.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.710+0000 7fdcc787e740 20 client.25525 trim_cache size 0 max 16384 2024-09-21T02:09:27.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.737+0000 7fdcc787e740 1 Processor -- start 2024-09-21T02:09:27.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.737+0000 7fdcc787e740 1 -- start start 2024-09-21T02:09:27.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1bee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b73c0 con 0x55bfbce12840 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b8f00 con 0x55bfbcdf1b60 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcc787e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b7a90 con 0x55bfbcde9980 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1bee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:39862/0 (socket says 172.21.15.16:39862) 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 -- 172.21.15.16:0/2813453832 learned_addr learned my addr 172.21.15.16:0/2813453832 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 -- 172.21.15.16:0/2813453832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 msgr2=0x55bfbd1bee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1bee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 -- 172.21.15.16:0/2813453832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 msgr2=0x55bfbd1c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.738+0000 7fdcb3fff640 1 -- 172.21.15.16:0/2813453832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfbd1b2ad0 con 0x55bfbcdf1b60 2024-09-21T02:09:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.739+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdca40aae30 tx=0x7fdca40e8440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f57e5c64a138f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.739+0000 7fdcb0ff9640 1 -- 172.21.15.16:0/2813453832 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40ad330 con 0x55bfbcdf1b60 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.739+0000 7fdcb0ff9640 1 -- 172.21.15.16:0/2813453832 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdca40b2470 con 0x55bfbcdf1b60 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.739+0000 7fdcb0ff9640 1 -- 172.21.15.16:0/2813453832 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40ad510 con 0x55bfbcdf1b60 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 -- 172.21.15.16:0/2813453832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 msgr2=0x55bfbd1d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdca40aae30 tx=0x7fdca40e8440 comp rx=0 tx=0).stop 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 -- 172.21.15.16:0/2813453832 shutdown_connections 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbce12840 0x55bfbd1c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1d1000 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 --2- 172.21.15.16:0/2813453832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbcde9980 0x55bfbd1bee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 -- 172.21.15.16:0/2813453832 >> 172.21.15.16:0/2813453832 conn(0x55bfbce06990 msgr2=0x55bfbd1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 -- 172.21.15.16:0/2813453832 shutdown_connections 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.740+0000 7fdcc787e740 1 -- 172.21.15.16:0/2813453832 wait complete. 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 -- start start 2024-09-21T02:09:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 Processor -- start 2024-09-21T02:09:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbce12840 0x55bfbd1ca160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b8830 con 0x55bfbcdf1b60 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b8160 con 0x55bfbcde9980 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.741+0000 7fdcc787e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1b53a0 con 0x55bfbce12840 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcc787e740 10 client.0 _ll_register_callbacks cb 0x7fdca4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:39874/0 (socket says 172.21.15.16:39874) 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 -- 172.21.15.16:0/1879329629 learned_addr learned my addr 172.21.15.16:0/1879329629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbb7fe640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbce12840 0x55bfbd1ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 -- 172.21.15.16:0/1879329629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbce12840 msgr2=0x55bfbd1ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbce12840 0x55bfbd1ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 -- 172.21.15.16:0/1879329629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 msgr2=0x55bfbd1c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 -- 172.21.15.16:0/1879329629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfbd1ca800 con 0x55bfbcde9980 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdcbaffd640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009900 tx=0x7fdcac009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6adb659d289dea7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.742+0000 7fdca27fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac05e060 con 0x55bfbcde9980 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fdcac0035f0 con 0x55bfbcde9980 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdca40b2470 con 0x55bfbcde9980 2024-09-21T02:09:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdcac065050 con 0x55bfbcde9980 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdcc787e740 10 client.25460 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bfbd242970 con 0x55bfbcde9980 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdca40b2470 con 0x55bfbcde9980 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.743+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fdcac058070 con 0x55bfbcde9980 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 10 client.25460 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 10 client.25460 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 20 client.25460 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcda6570 0x55bfbd245150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 4 client.25460 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x55bfbd12d590 con 0x55bfbcda6570 2024-09-21T02:09:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcda6570 0x55bfbd245150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdca27fc640 10 client.25460 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.744+0000 7fdcb3fff640 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcda6570 0x55bfbd245150 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8832de7d66c7c0d server_cookie=8edb97e7aa37bd7f in_seq=0 out_seq=0 2024-09-21T02:09:27.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdca27fc640 1 -- 172.21.15.16:0/1879329629 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55bfbd12d590 con 0x55bfbcda6570 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdca27fc640 10 client.25460 handle_command_reply: tid=0 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 client.25460 shutdown 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 msgr2=0x55bfbd1cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009900 tx=0x7fdcac009260 comp rx=0 tx=0).stop 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 shutdown_connections 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55bfbcda6570 0x55bfbd245150 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bfbce12840 0x55bfbd1ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bfbcdf1b60 0x55bfbd1c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 --2- 172.21.15.16:0/1879329629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55bfbcde9980 0x55bfbd1cd760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.789+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 >> 172.21.15.16:0/1879329629 conn(0x55bfbce06990 msgr2=0x55bfbd1ba630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.790+0000 7fdca27fc640 0 client.25460 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.790+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 shutdown_connections 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.790+0000 7fdcc787e740 1 -- 172.21.15.16:0/1879329629 wait complete. 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.790+0000 7fdcc787e740 20 client.25460 trim_cache size 0 max 16384 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 10, 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:09:27.792 INFO:teuthology.orchestra.run.smithi016.stdout: "b6a5b536": { 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 10, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.8, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:27.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 msgr2=0x7fdca80b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 0x7fdca80b8c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac000bc0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 msgr2=0x7fdcbc10fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdca400e9c0 tx=0x7fdca400ee80 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 shutdown_connections 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdca80b6700 0x7fdca80b8c10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc108d10 0x7fdcbc110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcbc1083c0 0x7fdcbc10fdd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 --2- 172.21.15.16:0/121164053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcbc102a80 0x7fdcbc1117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 >> 172.21.15.16:0/121164053 conn(0x7fdcbc079760 msgr2=0x7fdcbc101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 shutdown_connections 2024-09-21T02:09:27.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:27.791+0000 7fdcc130a640 1 -- 172.21.15.16:0/121164053 wait complete. 2024-09-21T02:09:28.825 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b6a5b536', quiesce took 4.5 seconds. Will release after: 23.3 2024-09-21T02:09:29.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:29 smithi123 ceph-mon[33453]: pgmap v422: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 143 MiB/s wr, 81 op/s 2024-09-21T02:09:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:29 smithi016 ceph-mon[31245]: pgmap v422: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 143 MiB/s wr, 81 op/s 2024-09-21T02:09:29.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:29 smithi161 ceph-mon[34061]: pgmap v422: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 143 MiB/s wr, 81 op/s 2024-09-21T02:09:31.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:31 smithi123 ceph-mon[33453]: pgmap v423: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 148 MiB/s wr, 86 op/s 2024-09-21T02:09:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:31 smithi016 ceph-mon[31245]: pgmap v423: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 148 MiB/s wr, 86 op/s 2024-09-21T02:09:31.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:31 smithi161 ceph-mon[34061]: pgmap v423: 129 pgs: 129 active+clean; 6.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 148 MiB/s wr, 86 op/s 2024-09-21T02:09:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:33 smithi123 ceph-mon[33453]: pgmap v424: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 129 MiB/s wr, 76 op/s 2024-09-21T02:09:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:33 smithi016 ceph-mon[31245]: pgmap v424: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 129 MiB/s wr, 76 op/s 2024-09-21T02:09:33.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:33 smithi161 ceph-mon[34061]: pgmap v424: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 129 MiB/s wr, 76 op/s 2024-09-21T02:09:35.466 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:35 smithi161 ceph-mon[34061]: pgmap v425: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 100 MiB/s wr, 63 op/s 2024-09-21T02:09:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:35 smithi123 ceph-mon[33453]: pgmap v425: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 100 MiB/s wr, 63 op/s 2024-09-21T02:09:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:35 smithi016 ceph-mon[31245]: pgmap v425: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 100 MiB/s wr, 63 op/s 2024-09-21T02:09:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:37 smithi123 ceph-mon[33453]: pgmap v426: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 62 MiB/s wr, 42 op/s 2024-09-21T02:09:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:37 smithi016 ceph-mon[31245]: pgmap v426: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 62 MiB/s wr, 42 op/s 2024-09-21T02:09:37.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:37 smithi161 ceph-mon[34061]: pgmap v426: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 62 MiB/s wr, 42 op/s 2024-09-21T02:09:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:39 smithi123 ceph-mon[33453]: pgmap v427: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 35 op/s 2024-09-21T02:09:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:39 smithi016 ceph-mon[31245]: pgmap v427: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 35 op/s 2024-09-21T02:09:39.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:39 smithi161 ceph-mon[34061]: pgmap v427: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 35 op/s 2024-09-21T02:09:40.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:40 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:09:40.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:40 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:09:40.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:40 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:09:40.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:40 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:09:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:40 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:09:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:40 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:09:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:40 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:09:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:40 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:09:40.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:40 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:09:40.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:40 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:09:40.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:40 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:09:40.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:40 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:09:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:41 smithi123 ceph-mon[33453]: pgmap v428: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 34 MiB/s wr, 29 op/s 2024-09-21T02:09:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:41 smithi016 ceph-mon[31245]: pgmap v428: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 34 MiB/s wr, 29 op/s 2024-09-21T02:09:41.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:41 smithi161 ceph-mon[34061]: pgmap v428: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 34 MiB/s wr, 29 op/s 2024-09-21T02:09:43.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:43 smithi123 ceph-mon[33453]: pgmap v429: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 MiB/s wr, 15 op/s 2024-09-21T02:09:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:43 smithi016 ceph-mon[31245]: pgmap v429: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 MiB/s wr, 15 op/s 2024-09-21T02:09:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:43 smithi161 ceph-mon[34061]: pgmap v429: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 MiB/s wr, 15 op/s 2024-09-21T02:09:45.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:45 smithi123 ceph-mon[33453]: pgmap v430: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1003 KiB/s wr, 12 op/s 2024-09-21T02:09:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:45 smithi016 ceph-mon[31245]: pgmap v430: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1003 KiB/s wr, 12 op/s 2024-09-21T02:09:45.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:45 smithi161 ceph-mon[34061]: pgmap v430: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1003 KiB/s wr, 12 op/s 2024-09-21T02:09:47.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:47 smithi123 ceph-mon[33453]: pgmap v431: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 492 KiB/s wr, 9 op/s 2024-09-21T02:09:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:47 smithi016 ceph-mon[31245]: pgmap v431: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 492 KiB/s wr, 9 op/s 2024-09-21T02:09:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:47 smithi161 ceph-mon[34061]: pgmap v431: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 492 KiB/s wr, 9 op/s 2024-09-21T02:09:49.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:49 smithi123 ceph-mon[33453]: pgmap v432: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 380 KiB/s wr, 9 op/s 2024-09-21T02:09:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:49 smithi016 ceph-mon[31245]: pgmap v432: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 380 KiB/s wr, 9 op/s 2024-09-21T02:09:49.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:49 smithi161 ceph-mon[34061]: pgmap v432: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 380 KiB/s wr, 9 op/s 2024-09-21T02:09:51.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:51 smithi123 ceph-mon[33453]: pgmap v433: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 78 KiB/s wr, 11 op/s 2024-09-21T02:09:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:51 smithi016 ceph-mon[31245]: pgmap v433: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 78 KiB/s wr, 11 op/s 2024-09-21T02:09:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:51 smithi161 ceph-mon[34061]: pgmap v433: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 78 KiB/s wr, 11 op/s 2024-09-21T02:09:52.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:09:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 -- 172.21.15.16:0/1733917635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 msgr2=0x7fafe4103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 --2- 172.21.15.16:0/1733917635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe4103f70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009900 tx=0x7fafe002a410 comp rx=0 tx=0).stop 2024-09-21T02:09:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 -- 172.21.15.16:0/1733917635 shutdown_connections 2024-09-21T02:09:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 --2- 172.21.15.16:0/1733917635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 0x7fafe4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 --2- 172.21.15.16:0/1733917635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 0x7fafe41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 --2- 172.21.15.16:0/1733917635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe4103f70 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 -- 172.21.15.16:0/1733917635 >> 172.21.15.16:0/1733917635 conn(0x7fafe407b850 msgr2=0x7fafe407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 -- 172.21.15.16:0/1733917635 shutdown_connections 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.913+0000 7fafebb75640 1 -- 172.21.15.16:0/1733917635 wait complete. 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 Processor -- start 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 -- start start 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 0x7fafe419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 0x7fafe41a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafe4113960 con 0x7fafe4103b60 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fafe41137e0 con 0x7fafe41094a0 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafebb75640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fafe4113ae0 con 0x7fafe4109df0 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafe9911640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafe9911640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40862/0 (socket says 172.21.15.16:40862) 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafe9911640 1 -- 172.21.15.16:0/4170770767 learned_addr learned my addr 172.21.15.16:0/4170770767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafea112640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 0x7fafe41a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.914+0000 7fafe9110640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 0x7fafe419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 -- 172.21.15.16:0/4170770767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 msgr2=0x7fafe41a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 0x7fafe41a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 -- 172.21.15.16:0/4170770767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 msgr2=0x7fafe419cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 0x7fafe419cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 -- 172.21.15.16:0/4170770767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe41a2460 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafe9911640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009230 tx=0x7fafe0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe003f030 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafe0003a80 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe41a26f0 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0035070 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.915+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe41a2c20 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.916+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe4103f70 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.918+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fafe0042030 con 0x7fafe4103b60 2024-09-21T02:09:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.919+0000 7fafdaffd640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 0x7fafd00b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.919+0000 7fafe9110640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 0x7fafd00b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.919+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7fafe0108590 con 0x7fafe4103b60 2024-09-21T02:09:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.920+0000 7fafe9110640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 0x7fafd00b8c40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005e10 tx=0x7fafd4005da0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:52.923+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fafe41a2c20 con 0x7fafe4103b60 2024-09-21T02:09:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.149+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafe419d6e0 con 0x7fafe4103b60 2024-09-21T02:09:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.150+0000 7fafdaffd640 1 -- 172.21.15.16:0/4170770767 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fafe419d6e0 con 0x7fafe4103b60 2024-09-21T02:09:53.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:09:53.151 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:09:53.152 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:09:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 msgr2=0x7fafd00b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 0x7fafd00b8c40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005e10 tx=0x7fafd4005da0 comp rx=0 tx=0).stop 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 msgr2=0x7fafe419c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009230 tx=0x7fafe0004270 comp rx=0 tx=0).stop 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 shutdown_connections 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fafd00b6730 0x7fafd00b8c40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafe4109df0 0x7fafe41a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafe41094a0 0x7fafe419cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 --2- 172.21.15.16:0/4170770767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafe4103b60 0x7fafe419c5b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 >> 172.21.15.16:0/4170770767 conn(0x7fafe407b850 msgr2=0x7fafe4107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 shutdown_connections 2024-09-21T02:09:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:53.152+0000 7fafebb75640 1 -- 172.21.15.16:0/4170770767 wait complete. 2024-09-21T02:09:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:53 smithi016 ceph-mon[31245]: pgmap v434: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T02:09:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:53 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4170770767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:53 smithi123 ceph-mon[33453]: pgmap v434: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T02:09:53.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:53 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4170770767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:53 smithi161 ceph-mon[34061]: pgmap v434: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T02:09:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:53 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4170770767' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:54.026 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id b6a5b536 --release --await' 2024-09-21T02:09:54.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id b6a5b536 --release --await 2024-09-21T02:09:54.886 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:55.274 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:55 smithi016 ceph-mon[31245]: pgmap v435: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 -- 172.21.15.16:0/2357126662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 msgr2=0x7f610c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 --2- 172.21.15.16:0/2357126662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c1098c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009900 tx=0x7f60fc02a410 comp rx=0 tx=0).stop 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 -- 172.21.15.16:0/2357126662 shutdown_connections 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 --2- 172.21.15.16:0/2357126662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 --2- 172.21.15.16:0/2357126662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c1098c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 --2- 172.21.15.16:0/2357126662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 -- 172.21.15.16:0/2357126662 >> 172.21.15.16:0/2357126662 conn(0x7f610c077c60 msgr2=0x7f610c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 -- 172.21.15.16:0/2357126662 shutdown_connections 2024-09-21T02:09:55.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.374+0000 7f6112b52640 1 -- 172.21.15.16:0/2357126662 wait complete. 2024-09-21T02:09:55.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.375+0000 7f6112b52640 1 Processor -- start 2024-09-21T02:09:55.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.375+0000 7f6112b52640 1 -- start start 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.375+0000 7f6112b52640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6112b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6112b52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6112b52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f610c113970 con 0x7f610c1094b0 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6112b52640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1137f0 con 0x7f610c109e00 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6112b52640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f610c113af0 con 0x7f610c101f70 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40880/0 (socket says 172.21.15.16:40880) 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 -- 172.21.15.16:0/679761634 learned_addr learned my addr 172.21.15.16:0/679761634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f61108ee640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f61110ef640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 -- 172.21.15.16:0/679761634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 msgr2=0x7f610c19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 -- 172.21.15.16:0/679761634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 msgr2=0x7f610c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 -- 172.21.15.16:0/679761634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c1a0330 con 0x7f610c1094b0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f61108ee640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f61110ef640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.376+0000 7f6103fff640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005940 tx=0x7f60fc005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.377+0000 7f6101ffb640 1 -- 172.21.15.16:0/679761634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc03f030 con 0x7f610c1094b0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.377+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c1a05c0 con 0x7f610c1094b0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.377+0000 7f6101ffb640 1 -- 172.21.15.16:0/679761634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60fc0043d0 con 0x7f610c1094b0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.377+0000 7f6101ffb640 1 -- 172.21.15.16:0/679761634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc035070 con 0x7f610c1094b0 2024-09-21T02:09:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.377+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1a6d80 con 0x7f610c1094b0 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.379+0000 7f61190c6740 1 Processor -- start 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 -- start start 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f6101ffb640 1 -- 172.21.15.16:0/679761634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f60fc042030 con 0x7f610c1094b0 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca9840 0x56464b070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56464af5bdc0 con 0x56464acca520 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56464afe81f0 con 0x56464aca1660 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61190c6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56464afd7620 con 0x56464aca9840 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40886/0 (socket says 172.21.15.16:40886) 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 -- 172.21.15.16:0/2368009974 learned_addr learned my addr 172.21.15.16:0/2368009974 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61108ee640 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f6103fff640 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca9840 0x56464b070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 -- 172.21.15.16:0/2368009974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca9840 msgr2=0x56464b070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca9840 0x56464b070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 -- 172.21.15.16:0/2368009974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 msgr2=0x56464b070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.380+0000 7f61110ef640 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b070a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f61110ef640 1 -- 172.21.15.16:0/2368009974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56464af9ffd0 con 0x56464acca520 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f61110ef640 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6104009fd0 tx=0x7f61040108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8376fbebe3512a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f60f2ffd640 1 -- 172.21.15.16:0/2368009974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6104011760 con 0x56464acca520 2024-09-21T02:09:55.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f60f2ffd640 1 -- 172.21.15.16:0/2368009974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6104011d90 con 0x56464acca520 2024-09-21T02:09:55.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f60f2ffd640 1 -- 172.21.15.16:0/2368009974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610401be50 con 0x56464acca520 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f6101ffb640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 0x7f60f40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.381+0000 7f61108ee640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 0x7f60f40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f6101ffb640 1 -- 172.21.15.16:0/679761634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f60fc107f20 con 0x7f610c1094b0 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 -- 172.21.15.16:0/2368009974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 msgr2=0x56464b070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61108ee640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 0x7f60f40b8b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x56464b06f190 tx=0x7f6108005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6104009fd0 tx=0x7f61040108e0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 -- 172.21.15.16:0/2368009974 shutdown_connections 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca9840 0x56464b070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 --2- 172.21.15.16:0/2368009974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464acca520 0x56464b070510 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 -- 172.21.15.16:0/2368009974 >> 172.21.15.16:0/2368009974 conn(0x56464acbe910 msgr2=0x56464b06e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 -- 172.21.15.16:0/2368009974 shutdown_connections 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.382+0000 7f61190c6740 1 -- 172.21.15.16:0/2368009974 wait complete. 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 -- start start 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 Processor -- start 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca1660 0x56464b07bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464acca520 0x56464b08c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56464af72300 con 0x56464aca9840 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x56464afdb430 con 0x56464acca520 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x56464ae38bb0 con 0x56464aca1660 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f61190c6740 10 client.0 _ll_register_callbacks cb 0x56464af8c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40890/0 (socket says 172.21.15.16:40890) 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.383+0000 7f6103fff640 1 -- 172.21.15.16:0/1689742679 learned_addr learned my addr 172.21.15.16:0/1689742679 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f61108ee640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca1660 0x56464b07bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f61110ef640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464acca520 0x56464b08c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 -- 172.21.15.16:0/1689742679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca1660 msgr2=0x56464b07bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca1660 0x56464b07bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 -- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464acca520 msgr2=0x56464b08c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464acca520 0x56464b08c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 -- 172.21.15.16:0/1689742679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56464af61780 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f6103fff640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0c9f70 tx=0x7f60fc0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613d615e0492eb29 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f60ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc049040 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6104011d90 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.384+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f60fc0d7130 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f60fc0cd770 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 10 client.15858 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56464ad35b00 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6104011d90 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f60fc0d2020 con 0x56464aca9840 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 10 client.15858 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 10 client.15858 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 20 client.15858 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac88ea0 0x56464b092350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 4 client.15858 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.385+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56464afea140 con 0x56464ac88ea0 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.386+0000 7f61108ee640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac88ea0 0x56464b092350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.386+0000 7f60ebfff640 10 client.15858 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.386+0000 7f61108ee640 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac88ea0 0x56464b092350 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7031c292e4f532d2 server_cookie=913347e6369fdaf7 in_seq=0 out_seq=0 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f60ebfff640 1 -- 172.21.15.16:0/1689742679 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56464afea140 con 0x56464ac88ea0 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f60ebfff640 10 client.15858 handle_command_reply: tid=0 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 client.15858 shutdown 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 msgr2=0x56464b07c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0c9f70 tx=0x7f60fc0095d0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 shutdown_connections 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac88ea0 0x56464b092350 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464acca520 0x56464b08c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b07c190 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f60ebfff640 0 client.15858 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 --2- 172.21.15.16:0/1689742679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464aca1660 0x56464b07bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 >> 172.21.15.16:0/1689742679 conn(0x56464acbe910 msgr2=0x56464b0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 shutdown_connections 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 1 -- 172.21.15.16:0/1689742679 wait complete. 2024-09-21T02:09:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.387+0000 7f61190c6740 20 client.15858 trim_cache size 0 max 16384 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 Processor -- start 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 -- start start 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 0x56464b071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b078e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1aeb90 con 0x56464aca1660 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b06d0 con 0x56464aca9840 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61190c6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1af260 con 0x56464acca520 2024-09-21T02:09:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61108ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61108ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40902/0 (socket says 172.21.15.16:40902) 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61108ee640 1 -- 172.21.15.16:0/125442624 learned_addr learned my addr 172.21.15.16:0/125442624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f6103fff640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 0x56464b071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.405+0000 7f61110ef640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b078e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 -- 172.21.15.16:0/125442624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 msgr2=0x56464b078e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b078e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 -- 172.21.15.16:0/125442624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 msgr2=0x56464b071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 0x56464b071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 -- 172.21.15.16:0/125442624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56464adac960 con 0x56464aca1660 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f6103fff640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 0x56464b071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f61108ee640 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6108009ae0 tx=0x7f6108016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661c0257594ce321 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f60f27fc640 1 -- 172.21.15.16:0/125442624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610802f070 con 0x56464aca1660 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f60f27fc640 1 -- 172.21.15.16:0/125442624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6108003c60 con 0x56464aca1660 2024-09-21T02:09:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.406+0000 7f60f27fc640 1 -- 172.21.15.16:0/125442624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108011450 con 0x56464aca1660 2024-09-21T02:09:55.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- 172.21.15.16:0/125442624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 msgr2=0x56464b08c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6108009ae0 tx=0x7f6108016000 comp rx=0 tx=0).stop 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- 172.21.15.16:0/125442624 shutdown_connections 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca9840 0x56464b071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 --2- 172.21.15.16:0/125442624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca1660 0x56464b08c1a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- 172.21.15.16:0/125442624 >> 172.21.15.16:0/125442624 conn(0x56464acbe910 msgr2=0x56464b071540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- 172.21.15.16:0/125442624 shutdown_connections 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- 172.21.15.16:0/125442624 wait complete. 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.407+0000 7f61190c6740 1 -- start start 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 Processor -- start 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b088790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b0851e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1b0000 con 0x56464aca9840 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1af930 con 0x56464aca1660 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f610c1acb70 con 0x56464acca520 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61190c6740 10 client.0 _ll_register_callbacks cb 0x7f60fc009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f6103fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40908/0 (socket says 172.21.15.16:40908) 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f6103fff640 1 -- 172.21.15.16:0/1746547949 learned_addr learned my addr 172.21.15.16:0/1746547949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61110ef640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b0851e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.408+0000 7f61108ee640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b088790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 -- 172.21.15.16:0/1746547949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 msgr2=0x56464b0851e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b0851e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 -- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 msgr2=0x56464b088790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b088790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 -- 172.21.15.16:0/1746547949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56464b085720 con 0x56464aca9840 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f6103fff640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0d7510 tx=0x7f60fc0cce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a58c7092b84d8ee server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:55.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f60ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc049030 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f60fc10b300 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6108003c60 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f60fc105730 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f61190c6740 10 client.15864 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.409+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56464b0fda80 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6108003c60 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f60fc0d3610 con 0x56464aca9840 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 10 client.15864 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 10 client.15864 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 20 client.15864 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac5e250 0x56464b100080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 4 client.15864 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b6a5b536", "release": true, "await": true}] 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b6a5b536", "release": true, "await": true}) -- 0x56464afe5bf0 con 0x56464ac5e250 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.410+0000 7f61108ee640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac5e250 0x56464b100080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.411+0000 7f61108ee640 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac5e250 0x56464b100080 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e56a5447b9f2a41 server_cookie=6e6a283e3d49156c in_seq=0 out_seq=0 2024-09-21T02:09:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.411+0000 7f60ebfff640 10 client.15864 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:55.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.416+0000 7f60ebfff640 1 -- 172.21.15.16:0/1746547949 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56464afe5bf0 con 0x56464ac5e250 2024-09-21T02:09:55.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.416+0000 7f60ebfff640 10 client.15864 handle_command_reply: tid=0 2024-09-21T02:09:55.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.416+0000 7f61190c6740 1 client.15864 shutdown 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.416+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 msgr2=0x56464b084ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.416+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0d7510 tx=0x7f60fc0cce30 comp rx=0 tx=0).stop 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 shutdown_connections 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x56464ac5e250 0x56464b100080 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56464acca520 0x56464b0851e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56464aca9840 0x56464b084ca0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 --2- 172.21.15.16:0/1746547949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56464aca1660 0x56464b088790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 >> 172.21.15.16:0/1746547949 conn(0x56464acbe910 msgr2=0x56464b07d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f60ebfff640 0 client.15864 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 shutdown_connections 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.417+0000 7f61190c6740 1 -- 172.21.15.16:0/1746547949 wait complete. 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.418+0000 7f61190c6740 20 client.15864 trim_cache size 0 max 16384 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:09:55.419 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 12, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "b6a5b536": { 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 12, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.8, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:09:55.420 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:55.421 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 msgr2=0x7f60f40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 0x7f60f40b8b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x56464b06f190 tx=0x7f6108005f90 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 msgr2=0x7f610c19a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005940 tx=0x7f60fc005e10 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 shutdown_connections 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f60f40b6680 0x7f60f40b8b90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f610c109e00 0x7f610c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610c1094b0 0x7f610c19a960 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 --2- 172.21.15.16:0/679761634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f610c101f70 0x7f610c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 >> 172.21.15.16:0/679761634 conn(0x7f610c077c60 msgr2=0x7f610c1055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:55.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 shutdown_connections 2024-09-21T02:09:55.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:55.419+0000 7f6112b52640 1 -- 172.21.15.16:0/679761634 wait complete. 2024-09-21T02:09:55.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:55 smithi123 ceph-mon[33453]: pgmap v435: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-21T02:09:55.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:55 smithi161 ceph-mon[34061]: pgmap v435: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-21T02:09:56.314 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b6a5b536', total seconds elapsed: 32.0 2024-09-21T02:09:56.314 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.8 2024-09-21T02:09:56.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:09:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 -- 172.21.15.16:0/3723371475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 msgr2=0x7f7c341087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 --2- 172.21.15.16:0/3723371475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c341087b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009900 tx=0x7f7c2802c600 comp rx=0 tx=0).stop 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 -- 172.21.15.16:0/3723371475 shutdown_connections 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 --2- 172.21.15.16:0/3723371475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 --2- 172.21.15.16:0/3723371475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c341087b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 --2- 172.21.15.16:0/3723371475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 -- 172.21.15.16:0/3723371475 >> 172.21.15.16:0/3723371475 conn(0x7f7c34079730 msgr2=0x7f7c34079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 -- 172.21.15.16:0/3723371475 shutdown_connections 2024-09-21T02:09:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.282+0000 7f7c3a652640 1 -- 172.21.15.16:0/3723371475 wait complete. 2024-09-21T02:09:57.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.283+0000 7f7c3a652640 1 Processor -- start 2024-09-21T02:09:57.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.283+0000 7f7c3a652640 1 -- start start 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c34198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34111710 con 0x7f7c34102a60 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34111590 con 0x7f7c341083a0 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c3a652640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34111890 con 0x7f7c34108cf0 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:57976/0 (socket says 172.21.15.16:57976) 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c34198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:57.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 -- 172.21.15.16:0/994538082 learned_addr learned my addr 172.21.15.16:0/994538082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c33fff640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 -- 172.21.15.16:0/994538082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 msgr2=0x7f7c34198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c34198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 -- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 msgr2=0x7f7c34198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c38bef640 1 -- 172.21.15.16:0/994538082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3419e0d0 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c337fe640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c34198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.284+0000 7f7c33fff640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c38bef640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00eb10 tx=0x7f7c2c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c004b20 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c3419e3c0 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c2c004cc0 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c341a4b20 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.285+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c010690 con 0x7f7c34108cf0 2024-09-21T02:09:57.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.286+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c34102e70 con 0x7f7c34108cf0 2024-09-21T02:09:57.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.288+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7c2c005020 con 0x7f7c34108cf0 2024-09-21T02:09:57.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.289+0000 7f7c317fa640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 0x7f7c1c0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:57.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.289+0000 7f7c33fff640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 0x7f7c1c0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:57.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.289+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f7c2c0e80d0 con 0x7f7c34108cf0 2024-09-21T02:09:57.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.290+0000 7f7c33fff640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 0x7f7c1c0b8c90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009c20 tx=0x7f7c24005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:57.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.293+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7c2c0ae250 con 0x7f7c34108cf0 2024-09-21T02:09:57.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:57 smithi016 ceph-mon[31245]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 685 KiB/s wr, 9 op/s 2024-09-21T02:09:57.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:57 smithi123 ceph-mon[33453]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 685 KiB/s wr, 9 op/s 2024-09-21T02:09:57.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:57 smithi161 ceph-mon[34061]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 685 KiB/s wr, 9 op/s 2024-09-21T02:09:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.682+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c3402e640 con 0x7f7c34108cf0 2024-09-21T02:09:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.683+0000 7f7c317fa640 1 -- 172.21.15.16:0/994538082 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f7c2c0b2630 con 0x7f7c34108cf0 2024-09-21T02:09:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:09:57.685 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:09:57.685 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 msgr2=0x7f7c1c0b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 0x7f7c1c0b8c90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009c20 tx=0x7f7c24005f90 comp rx=0 tx=0).stop 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 msgr2=0x7f7c3419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00eb10 tx=0x7f7c2c00c2d0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 shutdown_connections 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7c1c0b6780 0x7f7c1c0b8c90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c34108cf0 0x7f7c3419d9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c341083a0 0x7f7c34198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 --2- 172.21.15.16:0/994538082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c34102a60 0x7f7c34198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 >> 172.21.15.16:0/994538082 conn(0x7f7c34079730 msgr2=0x7f7c341044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 shutdown_connections 2024-09-21T02:09:57.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:57.686+0000 7f7c3a652640 1 -- 172.21.15.16:0/994538082 wait complete. 2024-09-21T02:09:58.403 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:58 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/994538082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:58.447 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0' 2024-09-21T02:09:58.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 147.8 --await-for 45.0 2024-09-21T02:09:58.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:58 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/994538082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:58.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:58 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/994538082' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:09:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:09:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 -- 172.21.15.16:0/1105651816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 msgr2=0x7f599410f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 --2- 172.21.15.16:0/1105651816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f599410f540 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f598c009ca0 tx=0x7f598c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 -- 172.21.15.16:0/1105651816 shutdown_connections 2024-09-21T02:09:59.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 --2- 172.21.15.16:0/1105651816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f599410f540 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 --2- 172.21.15.16:0/1105651816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f59941087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 --2- 172.21.15.16:0/1105651816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f5994102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 -- 172.21.15.16:0/1105651816 >> 172.21.15.16:0/1105651816 conn(0x7f5994079760 msgr2=0x7f5994079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 -- 172.21.15.16:0/1105651816 shutdown_connections 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.429+0000 7f599ab7d640 1 -- 172.21.15.16:0/1105651816 wait complete. 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.430+0000 7f599ab7d640 1 Processor -- start 2024-09-21T02:09:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.430+0000 7f599ab7d640 1 -- start start 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f599410ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f5994110510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5994113980 con 0x7f5994108d10 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5994113800 con 0x7f59941083c0 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599ab7d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5994113b00 con 0x7f5994102a80 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599911a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599911a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46798/0 (socket says 172.21.15.16:46798) 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f599911a640 1 -- 172.21.15.16:0/757084738 learned_addr learned my addr 172.21.15.16:0/757084738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.431+0000 7f5993fff640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f5994110510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 -- 172.21.15.16:0/757084738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 msgr2=0x7f599410ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5998919640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f599410ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f599410ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 -- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 msgr2=0x7f5994110a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 -- 172.21.15.16:0/757084738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59941a8690 con 0x7f59941083c0 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5998919640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f599410ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f599911a640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5993fff640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f5994110510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9c0 tx=0x7f598000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5991ffb640 1 -- 172.21.15.16:0/757084738 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598000ccc0 con 0x7f59941083c0 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.432+0000 7f5991ffb640 1 -- 172.21.15.16:0/757084738 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59800108c0 con 0x7f59941083c0 2024-09-21T02:09:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.433+0000 7f5991ffb640 1 -- 172.21.15.16:0/757084738 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980018750 con 0x7f59941083c0 2024-09-21T02:09:59.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.433+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59941a8980 con 0x7f59941083c0 2024-09-21T02:09:59.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.434+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59941a8e70 con 0x7f59941083c0 2024-09-21T02:09:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.437+0000 7f59a10f1740 1 Processor -- start 2024-09-21T02:09:59.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 -- start start 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f5991ffb640 1 -- 172.21.15.16:0/757084738 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f5980010430 con 0x7f59941083c0 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a3162080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5564a30d7b90 con 0x5564a2d91980 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5564a30c6fc0 con 0x5564a2dba840 2024-09-21T02:09:59.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f59a10f1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5564a30cadd0 con 0x5564a2d99b60 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f5998919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f599911a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f5998919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46808/0 (socket says 172.21.15.16:46808) 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f599911a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:40852/0 (socket says 172.21.15.16:40852) 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f5998919640 1 -- 172.21.15.16:0/3935518582 learned_addr learned my addr 172.21.15.16:0/3935518582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.438+0000 7f5993fff640 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a3162080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 -- 172.21.15.16:0/3935518582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 msgr2=0x5564a3162080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a3162080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 -- 172.21.15.16:0/3935518582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 msgr2=0x5564a3161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 -- 172.21.15.16:0/3935518582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a304e060 con 0x5564a2dba840 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f5998919640 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.439+0000 7f599911a640 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f598c006010 tx=0x7f598c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3181d5abbfdf7dcd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f5991ffb640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 0x7f59840b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f597effd640 1 -- 172.21.15.16:0/3935518582 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c031b80 con 0x5564a2dba840 2024-09-21T02:09:59.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f597effd640 1 -- 172.21.15.16:0/3935518582 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f598c036030 con 0x5564a2dba840 2024-09-21T02:09:59.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f597effd640 1 -- 172.21.15.16:0/3935518582 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c004650 con 0x5564a2dba840 2024-09-21T02:09:59.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f5991ffb640 1 -- 172.21.15.16:0/757084738 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f59800e8030 con 0x7f59941083c0 2024-09-21T02:09:59.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.440+0000 7f5998919640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 0x7f59840b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 -- 172.21.15.16:0/3935518582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 msgr2=0x5564a3161600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f598c006010 tx=0x7f598c009920 comp rx=0 tx=0).stop 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f5998919640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 0x7f59840b8c10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5564a3160280 tx=0x7f5988005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 -- 172.21.15.16:0/3935518582 shutdown_connections 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a3162080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3161b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 --2- 172.21.15.16:0/3935518582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3161600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.441+0000 7f59a10f1740 1 -- 172.21.15.16:0/3935518582 >> 172.21.15.16:0/3935518582 conn(0x5564a2dae990 msgr2=0x5564a315f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.442+0000 7f59a10f1740 1 -- 172.21.15.16:0/3935518582 shutdown_connections 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.442+0000 7f59a10f1740 1 -- 172.21.15.16:0/3935518582 wait complete. 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.442+0000 7f59a10f1740 1 -- start start 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.442+0000 7f59a10f1740 1 Processor -- start 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a316d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3178f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5564a2f287a0 con 0x5564a2d91980 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5564a306c0a0 con 0x5564a2dba840 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5564a316fc00 con 0x5564a2d99b60 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f5993fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.443+0000 7f59a10f1740 10 client.0 _ll_register_callbacks cb 0x5564a30493c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:58018/0 (socket says 172.21.15.16:58018) 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 -- 172.21.15.16:0/1862755321 learned_addr learned my addr 172.21.15.16:0/1862755321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:59.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5998919640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a316d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 -- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 msgr2=0x5564a3178f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3178f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 -- 172.21.15.16:0/1862755321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 msgr2=0x5564a316d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a316d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5993fff640 1 -- 172.21.15.16:0/1862755321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a305a880 con 0x5564a2d99b60 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.444+0000 7f5998919640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a316d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:59.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.445+0000 7f5993fff640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59800aad80 tx=0x7f59800a9a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cc2ec474f41a4cb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.445+0000 7f5973fff640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800ad280 con 0x5564a2d99b60 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f59800ea2d0 con 0x5564a2d99b60 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598c036030 con 0x5564a2d99b60 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5980010720 con 0x5564a2d99b60 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f59a10f1740 10 client.25570 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:59.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.446+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564a2e34b90 con 0x5564a2d99b60 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f598c036030 con 0x5564a2d99b60 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f59800a9460 con 0x5564a2d99b60 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 10 client.25570 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 10 client.25570 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 20 client.25570 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:59.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d791c0 0x5564a317f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 4 client.25570 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564a30d9ae0 con 0x5564a2d791c0 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.447+0000 7f5998919640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d791c0 0x5564a317f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.448+0000 7f5998919640 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d791c0 0x5564a317f4b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71dac421db9bd4dc server_cookie=85da548e6a63b189 in_seq=0 out_seq=0 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.448+0000 7f5973fff640 10 client.25570 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:59.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f5973fff640 1 -- 172.21.15.16:0/1862755321 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5564a30d9ae0 con 0x5564a2d791c0 2024-09-21T02:09:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f5973fff640 10 client.25570 handle_command_reply: tid=0 2024-09-21T02:09:59.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 client.25570 shutdown 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 msgr2=0x5564a316d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f59800aad80 tx=0x7f59800a9a80 comp rx=0 tx=0).stop 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 shutdown_connections 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d791c0 0x5564a317f4b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2dba840 0x5564a3178f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2d99b60 0x5564a316d7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 --2- 172.21.15.16:0/1862755321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a316d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 >> 172.21.15.16:0/1862755321 conn(0x5564a2dae990 msgr2=0x5564a31647e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.449+0000 7f5973fff640 0 client.25570 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.450+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 shutdown_connections 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.450+0000 7f59a10f1740 1 -- 172.21.15.16:0/1862755321 wait complete. 2024-09-21T02:09:59.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.450+0000 7f59a10f1740 20 client.25570 trim_cache size 0 max 16384 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.475+0000 7f59a10f1740 1 Processor -- start 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.475+0000 7f59a10f1740 1 -- start start 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.475+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d91980 0x5564a31687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.475+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.475+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a316a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f59a10f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b3000 con 0x5564a2d99b60 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f59a10f1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b4b40 con 0x5564a2d91980 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f59a10f1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b36d0 con 0x5564a2dba840 2024-09-21T02:09:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f5993fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f5993fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46830/0 (socket says 172.21.15.16:46830) 2024-09-21T02:09:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f5993fff640 1 -- 172.21.15.16:0/2063915642 learned_addr learned my addr 172.21.15.16:0/2063915642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a316a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f5998919640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d91980 0x5564a31687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 -- 172.21.15.16:0/2063915642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d91980 msgr2=0x5564a31687f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d91980 0x5564a31687f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 -- 172.21.15.16:0/2063915642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 msgr2=0x5564a3162d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 -- 172.21.15.16:0/2063915642 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a315aad0 con 0x5564a2dba840 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f5993fff640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.476+0000 7f599911a640 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a316a230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598c009bf0 tx=0x7f598c02a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12b1e85b715451c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.477+0000 7f597e7fc640 1 -- 172.21.15.16:0/2063915642 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c031a40 con 0x5564a2dba840 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.477+0000 7f597e7fc640 1 -- 172.21.15.16:0/2063915642 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f598c036070 con 0x5564a2dba840 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.477+0000 7f597e7fc640 1 -- 172.21.15.16:0/2063915642 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c0044a0 con 0x5564a2dba840 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- 172.21.15.16:0/2063915642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 msgr2=0x5564a316a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a316a230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598c009bf0 tx=0x7f598c02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- 172.21.15.16:0/2063915642 shutdown_connections 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a316a230 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d99b60 0x5564a3162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 --2- 172.21.15.16:0/2063915642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d91980 0x5564a31687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- 172.21.15.16:0/2063915642 >> 172.21.15.16:0/2063915642 conn(0x5564a2dae990 msgr2=0x5564a3162630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- 172.21.15.16:0/2063915642 shutdown_connections 2024-09-21T02:09:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- 172.21.15.16:0/2063915642 wait complete. 2024-09-21T02:09:59.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.478+0000 7f59a10f1740 1 -- start start 2024-09-21T02:09:59.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 Processor -- start 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d99b60 0x5564a31719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a3171f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b4470 con 0x5564a2d91980 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b3da0 con 0x5564a2d99b60 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59941b0fe0 con 0x5564a2dba840 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f59a10f1740 10 client.0 _ll_register_callbacks cb 0x7f59800eb690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f5998919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f5998919640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46844/0 (socket says 172.21.15.16:46844) 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f5998919640 1 -- 172.21.15.16:0/216306655 learned_addr learned my addr 172.21.15.16:0/216306655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.479+0000 7f599911a640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a3171f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 -- 172.21.15.16:0/216306655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 msgr2=0x5564a3171f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a3171f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 -- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d99b60 msgr2=0x5564a31719c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d99b60 0x5564a31719c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 -- 172.21.15.16:0/216306655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a3172540 con 0x5564a2d91980 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5998919640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x5564a317fc50 tx=0x7f59880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f540d42991525e33 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:09:59.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.480+0000 7f5973fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.482+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988036060 con 0x5564a2d91980 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.482+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f59880049b0 con 0x5564a2d91980 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598c036070 con 0x5564a2d91980 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5988005690 con 0x5564a2d91980 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 10 client.15888 fetch_fsmap learned FSMap version 28 2024-09-21T02:09:59.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564a31eace0 con 0x5564a2d91980 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f598c036070 con 0x5564a2d91980 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f598800f040 con 0x5564a2d91980 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 10 client.15888 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 10 client.15888 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 20 client.15888 populate_metadata read hostname 'smithi016' 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.483+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d4e570 0x5564a31ed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.484+0000 7f59a10f1740 4 client.15888 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}] 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.484+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 147.8, "await_for": 45.0}) -- 0x5564a30d5590 con 0x5564a2d4e570 2024-09-21T02:09:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.484+0000 7f5993fff640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d4e570 0x5564a31ed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:09:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.485+0000 7f5973fff640 10 client.15888 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.485+0000 7f5993fff640 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d4e570 0x5564a31ed4f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a7f3a4e839ecc16 server_cookie=80659a111b378b29 in_seq=0 out_seq=0 2024-09-21T02:09:59.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:09:59 smithi123 ceph-mon[33453]: pgmap v437: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 12 op/s 2024-09-21T02:09:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.526+0000 7f5973fff640 1 -- 172.21.15.16:0/216306655 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5564a30d5590 con 0x5564a2d4e570 2024-09-21T02:09:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.526+0000 7f5973fff640 10 client.15888 handle_command_reply: tid=0 2024-09-21T02:09:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.526+0000 7f59a10f1740 1 client.15888 shutdown 2024-09-21T02:09:59.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:09:59 smithi016 ceph-mon[31245]: pgmap v437: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 12 op/s 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 msgr2=0x5564a3175500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x5564a317fc50 tx=0x7f59880046c0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 shutdown_connections 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5564a2d4e570 0x5564a31ed4f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5564a2dba840 0x5564a3171f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5564a2d99b60 0x5564a31719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 --2- 172.21.15.16:0/216306655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5564a2d91980 0x5564a3175500 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 >> 172.21.15.16:0/216306655 conn(0x5564a2dae990 msgr2=0x5564a316b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f5973fff640 0 client.15888 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 shutdown_connections 2024-09-21T02:09:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.527+0000 7f59a10f1740 1 -- 172.21.15.16:0/216306655 wait complete. 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.528+0000 7f59a10f1740 20 client.15888 trim_cache size 0 max 16384 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 14, 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "2a4c270d": { 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 14, 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:09:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.8, 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.528+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 msgr2=0x7f59840b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.528+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 0x7f59840b8c10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5564a3160280 tx=0x7f5988005f90 comp rx=0 tx=0).stop 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 msgr2=0x7f5994110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f5994110510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9c0 tx=0x7f598000ee80 comp rx=0 tx=0).stop 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 shutdown_connections 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f59840b6700 0x7f59840b8c10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5994108d10 0x7f5994110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59941083c0 0x7f5994110510 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 --2- 172.21.15.16:0/757084738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5994102a80 0x7f599410ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:09:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 >> 172.21.15.16:0/757084738 conn(0x7f5994079760 msgr2=0x7f5994104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:09:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 shutdown_connections 2024-09-21T02:09:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:09:59.529+0000 7f599ab7d640 1 -- 172.21.15.16:0/757084738 wait complete. 2024-09-21T02:09:59.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:09:59 smithi161 ceph-mon[34061]: pgmap v437: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 12 op/s 2024-09-21T02:10:00.232 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2a4c270d', quiesce took 3.9 seconds. Will release after: 23.900000000000002 2024-09-21T02:10:00.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:00 smithi123 ceph-mon[33453]: overall HEALTH_OK 2024-09-21T02:10:00.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:00 smithi016 ceph-mon[31245]: overall HEALTH_OK 2024-09-21T02:10:00.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:00 smithi161 ceph-mon[34061]: overall HEALTH_OK 2024-09-21T02:10:01.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:01 smithi123 ceph-mon[33453]: pgmap v438: 129 pgs: 129 active+clean; 7.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-09-21T02:10:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:01 smithi016 ceph-mon[31245]: pgmap v438: 129 pgs: 129 active+clean; 7.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-09-21T02:10:01.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:01 smithi161 ceph-mon[34061]: pgmap v438: 129 pgs: 129 active+clean; 7.0 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-09-21T02:10:03.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:03 smithi123 ceph-mon[33453]: pgmap v439: 129 pgs: 129 active+clean; 7.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:10:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:03 smithi016 ceph-mon[31245]: pgmap v439: 129 pgs: 129 active+clean; 7.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:10:03.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:03 smithi161 ceph-mon[34061]: pgmap v439: 129 pgs: 129 active+clean; 7.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-09-21T02:10:05.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:05 smithi123 ceph-mon[33453]: pgmap v440: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 51 op/s 2024-09-21T02:10:05.649 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:05 smithi016 ceph-mon[31245]: pgmap v440: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 51 op/s 2024-09-21T02:10:05.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:05 smithi161 ceph-mon[34061]: pgmap v440: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 51 op/s 2024-09-21T02:10:07.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:07 smithi123 ceph-mon[33453]: pgmap v441: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 49 MiB/s wr, 49 op/s 2024-09-21T02:10:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:07 smithi016 ceph-mon[31245]: pgmap v441: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 49 MiB/s wr, 49 op/s 2024-09-21T02:10:07.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:07 smithi161 ceph-mon[34061]: pgmap v441: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 49 MiB/s wr, 49 op/s 2024-09-21T02:10:09.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:09 smithi016 ceph-mon[31245]: pgmap v442: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 48 MiB/s wr, 48 op/s 2024-09-21T02:10:09.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:09 smithi161 ceph-mon[34061]: pgmap v442: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 48 MiB/s wr, 48 op/s 2024-09-21T02:10:09.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:09 smithi123 ceph-mon[33453]: pgmap v442: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 48 MiB/s wr, 48 op/s 2024-09-21T02:10:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:11 smithi016 ceph-mon[31245]: pgmap v443: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 MiB/s wr, 48 op/s 2024-09-21T02:10:11.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:11 smithi161 ceph-mon[34061]: pgmap v443: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 MiB/s wr, 48 op/s 2024-09-21T02:10:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:11 smithi123 ceph-mon[33453]: pgmap v443: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 MiB/s wr, 48 op/s 2024-09-21T02:10:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:13 smithi016 ceph-mon[31245]: pgmap v444: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 MiB/s wr, 36 op/s 2024-09-21T02:10:13.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:13 smithi161 ceph-mon[34061]: pgmap v444: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 MiB/s wr, 36 op/s 2024-09-21T02:10:13.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:13 smithi123 ceph-mon[33453]: pgmap v444: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 MiB/s wr, 36 op/s 2024-09-21T02:10:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:15 smithi016 ceph-mon[31245]: pgmap v445: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.2 MiB/s wr, 31 op/s 2024-09-21T02:10:15.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:15 smithi161 ceph-mon[34061]: pgmap v445: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.2 MiB/s wr, 31 op/s 2024-09-21T02:10:15.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:15 smithi123 ceph-mon[33453]: pgmap v445: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.2 MiB/s wr, 31 op/s 2024-09-21T02:10:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:17 smithi016 ceph-mon[31245]: pgmap v446: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 561 KiB/s wr, 9 op/s 2024-09-21T02:10:17.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:17 smithi161 ceph-mon[34061]: pgmap v446: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 561 KiB/s wr, 9 op/s 2024-09-21T02:10:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:17 smithi123 ceph-mon[33453]: pgmap v446: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 561 KiB/s wr, 9 op/s 2024-09-21T02:10:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:19 smithi016 ceph-mon[31245]: pgmap v447: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 476 KiB/s wr, 9 op/s 2024-09-21T02:10:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:19 smithi161 ceph-mon[34061]: pgmap v447: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 476 KiB/s wr, 9 op/s 2024-09-21T02:10:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:19 smithi123 ceph-mon[33453]: pgmap v447: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 476 KiB/s wr, 9 op/s 2024-09-21T02:10:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:21 smithi016 ceph-mon[31245]: pgmap v448: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 11 op/s 2024-09-21T02:10:21.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:21 smithi161 ceph-mon[34061]: pgmap v448: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 11 op/s 2024-09-21T02:10:21.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:21 smithi123 ceph-mon[33453]: pgmap v448: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 11 op/s 2024-09-21T02:10:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:23 smithi016 ceph-mon[31245]: pgmap v449: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:23.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:23 smithi161 ceph-mon[34061]: pgmap v449: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:23.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:23 smithi123 ceph-mon[33453]: pgmap v449: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:24.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:10:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:10:24.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 -- 172.21.15.16:0/3152693611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a81094f0 msgr2=0x7f10a8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:25.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3152693611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a81094f0 0x7f10a8109900 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1098009900 tx=0x7f109802a410 comp rx=0 tx=0).stop 2024-09-21T02:10:25.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 -- 172.21.15.16:0/3152693611 shutdown_connections 2024-09-21T02:10:25.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3152693611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 0x7f10a8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3152693611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a81094f0 0x7f10a8109900 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3152693611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a8103bb0 0x7f10a8103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 -- 172.21.15.16:0/3152693611 >> 172.21.15.16:0/3152693611 conn(0x7f10a80ff600 msgr2=0x7f10a8101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 -- 172.21.15.16:0/3152693611 shutdown_connections 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.918+0000 7f10b02fd640 1 -- 172.21.15.16:0/3152693611 wait complete. 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 Processor -- start 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 -- start start 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a81094f0 0x7f10a819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 0x7f10a819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81139b0 con 0x7f10a8103bb0 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8113830 con 0x7f10a8109e40 2024-09-21T02:10:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10b02fd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8113b30 con 0x7f10a81094f0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.919+0000 7f10ae099640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58670/0 (socket says 172.21.15.16:58670) 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 -- 172.21.15.16:0/3124721400 learned_addr learned my addr 172.21.15.16:0/3124721400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ad898640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a81094f0 0x7f10a819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae89a640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 0x7f10a819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 -- 172.21.15.16:0/3124721400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a81094f0 msgr2=0x7f10a819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a81094f0 0x7f10a819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 -- 172.21.15.16:0/3124721400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 msgr2=0x7f10a819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 0x7f10a819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 -- 172.21.15.16:0/3124721400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a81a02b0 con 0x7f10a8103bb0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10ae099640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f10a400c920 tx=0x7f10a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4007c70 con 0x7f10a8103bb0 2024-09-21T02:10:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10a4007e10 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a81a05a0 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4018710 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.920+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a81a6dc0 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.921+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10a8103fc0 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.923+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f10a400f450 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.924+0000 7f109f7fe640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 0x7f10940b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.924+0000 7f10ad898640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 0x7f10940b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.924+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f10a40e7d90 con 0x7f10a8103bb0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.925+0000 7f10ad898640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 0x7f10940b8c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1098005940 tx=0x7f10980058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:24.928+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f10a4014030 con 0x7f10a8103bb0 2024-09-21T02:10:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.160+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10a819b530 con 0x7f10a8103bb0 2024-09-21T02:10:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.161+0000 7f109f7fe640 1 -- 172.21.15.16:0/3124721400 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f10a40ae080 con 0x7f10a8103bb0 2024-09-21T02:10:25.251 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:10:25.251 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:10:25.251 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 msgr2=0x7f10940b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 0x7f10940b8c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1098005940 tx=0x7f10980058d0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 msgr2=0x7f10a819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f10a400c920 tx=0x7f10a400cde0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 shutdown_connections 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10940b6730 0x7f10940b8c40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10a8109e40 0x7f10a819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10a81094f0 0x7f10a819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 --2- 172.21.15.16:0/3124721400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10a8103bb0 0x7f10a819a400 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 >> 172.21.15.16:0/3124721400 conn(0x7f10a80ff600 msgr2=0x7f10a8101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 shutdown_connections 2024-09-21T02:10:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:25.163+0000 7f10b02fd640 1 -- 172.21.15.16:0/3124721400 wait complete. 2024-09-21T02:10:25.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:25 smithi016 ceph-mon[31245]: pgmap v450: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:10:25.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:25 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3124721400' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:10:25.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:25 smithi161 ceph-mon[34061]: pgmap v450: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:10:25.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:25 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3124721400' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:10:25.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:25 smithi123 ceph-mon[33453]: pgmap v450: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:10:25.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:25 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3124721400' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:10:26.062 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 2a4c270d --release --await' 2024-09-21T02:10:26.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 2a4c270d --release --await 2024-09-21T02:10:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:10:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 -- 172.21.15.16:0/3312396776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c1094a0 msgr2=0x7f102c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 --2- 172.21.15.16:0/3312396776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c1094a0 0x7f102c1098b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f101c009900 tx=0x7f101c02a410 comp rx=0 tx=0).stop 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 -- 172.21.15.16:0/3312396776 shutdown_connections 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 --2- 172.21.15.16:0/3312396776 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c109df0 0x7f102c110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 --2- 172.21.15.16:0/3312396776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c1094a0 0x7f102c1098b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 --2- 172.21.15.16:0/3312396776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 0x7f102c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 -- 172.21.15.16:0/3312396776 >> 172.21.15.16:0/3312396776 conn(0x7f102c07b850 msgr2=0x7f102c07bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 -- 172.21.15.16:0/3312396776 shutdown_connections 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.853+0000 7f10341a4640 1 -- 172.21.15.16:0/3312396776 wait complete. 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 Processor -- start 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 -- start start 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 0x7f102c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c1094a0 0x7f102c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f102c113960 con 0x7f102c109df0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1137e0 con 0x7f102c103b60 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f10341a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f102c113ae0 con 0x7f102c1094a0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f1032741640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f1032741640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58700/0 (socket says 172.21.15.16:58700) 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.854+0000 7f1032741640 1 -- 172.21.15.16:0/4218074921 learned_addr learned my addr 172.21.15.16:0/4218074921 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f103173f640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c1094a0 0x7f102c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1031f40640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 0x7f102c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 -- 172.21.15.16:0/4218074921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c1094a0 msgr2=0x7f102c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c1094a0 0x7f102c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 -- 172.21.15.16:0/4218074921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 msgr2=0x7f102c075710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 0x7f102c075710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 -- 172.21.15.16:0/4218074921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c1a8730 con 0x7f102c109df0 2024-09-21T02:10:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1032741640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f102400eb10 tx=0x7f102400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1022ffd640 1 -- 172.21.15.16:0/4218074921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024004b20 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102c1a8a20 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1022ffd640 1 -- 172.21.15.16:0/4218074921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1024004cc0 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f1022ffd640 1 -- 172.21.15.16:0/4218074921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024010690 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.855+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102c1a8f30 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.858+0000 7f103a718740 1 Processor -- start 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.858+0000 7f1022ffd640 1 -- 172.21.15.16:0/4218074921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f1024005020 con 0x7f102c109df0 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.858+0000 7f103a718740 1 -- start start 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd6bddc0 con 0x55cfdd403660 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd74a1f0 con 0x55cfdd42c520 2024-09-21T02:10:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103a718740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd739620 con 0x55cfdd40b840 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58702/0 (socket says 172.21.15.16:58702) 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 -- 172.21.15.16:0/853647954 learned_addr learned my addr 172.21.15.16:0/853647954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103173f640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103173f640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1022ffd640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 0x7f10180b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 -- 172.21.15.16:0/853647954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 msgr2=0x55cfdd7d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 -- 172.21.15.16:0/853647954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 msgr2=0x55cfdd7d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1031f40640 1 -- 172.21.15.16:0/853647954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfdd701fd0 con 0x55cfdd403660 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f103173f640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.859+0000 7f1032741640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 0x7f10180b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.860+0000 7f1031f40640 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1028009c90 tx=0x7f1028011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7774eba3dc62cc2f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.860+0000 7f1022ffd640 1 -- 172.21.15.16:0/4218074921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f10240e7fc0 con 0x7f102c109df0 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.860+0000 7f1017fff640 1 -- 172.21.15.16:0/853647954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028005220 con 0x55cfdd403660 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.860+0000 7f1017fff640 1 -- 172.21.15.16:0/853647954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1028004c30 con 0x55cfdd403660 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.860+0000 7f1017fff640 1 -- 172.21.15.16:0/853647954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028019690 con 0x55cfdd403660 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- 172.21.15.16:0/853647954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 msgr2=0x55cfdd7d2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f1032741640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 0x7f10180b8b90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f10240e59a0 tx=0x7f10240aa1d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1028009c90 tx=0x7f1028011810 comp rx=0 tx=0).stop 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- 172.21.15.16:0/853647954 shutdown_connections 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7d2a50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 --2- 172.21.15.16:0/853647954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- 172.21.15.16:0/853647954 >> 172.21.15.16:0/853647954 conn(0x55cfdd420910 msgr2=0x55cfdd7d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- 172.21.15.16:0/853647954 shutdown_connections 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- 172.21.15.16:0/853647954 wait complete. 2024-09-21T02:10:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 -- start start 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.861+0000 7f103a718740 1 Processor -- start 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd40b840 0x55cfdd7de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd42c520 0x55cfdd7e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd6d4300 con 0x55cfdd403660 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd73d430 con 0x55cfdd40b840 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cfdd59abb0 con 0x55cfdd42c520 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103a718740 10 client.0 _ll_register_callbacks cb 0x55cfdd6ee120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58708/0 (socket says 172.21.15.16:58708) 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 -- 172.21.15.16:0/3891434214 learned_addr learned my addr 172.21.15.16:0/3891434214 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd42c520 0x55cfdd7e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f1031f40640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd40b840 0x55cfdd7de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 -- 172.21.15.16:0/3891434214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd42c520 msgr2=0x55cfdd7e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd42c520 0x55cfdd7e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 -- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd40b840 msgr2=0x55cfdd7de0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd40b840 0x55cfdd7de0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.862+0000 7f103173f640 1 -- 172.21.15.16:0/3891434214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfdd6c3780 con 0x55cfdd403660 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103173f640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f101c0057c0 tx=0x7f101c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf2696a5407be7f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:10:26.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c041070 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f101c031d30 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1028004c30 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f101c052050 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 10 client.15906 fetch_fsmap learned FSMap version 28 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cfdd497b00 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1028004c30 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f101c003ae0 con 0x55cfdd403660 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 10 client.15906 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 10 client.15906 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:10:26.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 20 client.15906 populate_metadata read hostname 'smithi016' 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3eaea0 0x55cfdd7efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.863+0000 7f103a718740 4 client.15906 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.864+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfdd74c140 con 0x55cfdd3eaea0 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.864+0000 7f1031f40640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3eaea0 0x55cfdd7efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.864+0000 7f10157fa640 10 client.15906 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.864+0000 7f1031f40640 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3eaea0 0x55cfdd7efee0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceb839520698ec3c server_cookie=f5153948b4777ca in_seq=0 out_seq=0 2024-09-21T02:10:26.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f10157fa640 1 -- 172.21.15.16:0/3891434214 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cfdd74c140 con 0x55cfdd3eaea0 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f10157fa640 10 client.15906 handle_command_reply: tid=0 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 client.15906 shutdown 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 msgr2=0x55cfdd7ddb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f101c0057c0 tx=0x7f101c02a410 comp rx=0 tx=0).stop 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 shutdown_connections 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3eaea0 0x55cfdd7efee0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd42c520 0x55cfdd7e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd40b840 0x55cfdd7de0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f10157fa640 0 client.15906 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 --2- 172.21.15.16:0/3891434214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7ddb90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 >> 172.21.15.16:0/3891434214 conn(0x55cfdd420910 msgr2=0x55cfdd7d3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 shutdown_connections 2024-09-21T02:10:26.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 1 -- 172.21.15.16:0/3891434214 wait complete. 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.865+0000 7f103a718740 20 client.15906 trim_cache size 0 max 16384 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 Processor -- start 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 -- start start 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd403660 0x55cfdd7e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b3070 con 0x55cfdd40b840 2024-09-21T02:10:26.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b4bb0 con 0x55cfdd42c520 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f103a718740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b3740 con 0x55cfdd403660 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f1031f40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.883+0000 7f1031f40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58712/0 (socket says 172.21.15.16:58712) 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 -- 172.21.15.16:0/2105738138 learned_addr learned my addr 172.21.15.16:0/2105738138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f103173f640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd403660 0x55cfdd7e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f103173f640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 -- 172.21.15.16:0/2105738138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd403660 msgr2=0x55cfdd7e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd403660 0x55cfdd7e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 -- 172.21.15.16:0/2105738138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 msgr2=0x55cfdd7f56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7f56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 -- 172.21.15.16:0/2105738138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfdd50e960 con 0x55cfdd40b840 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f103173f640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f1031f40640 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1028009230 tx=0x7f1028005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0e4073731d7e1b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f10177fe640 1 -- 172.21.15.16:0/2105738138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028004250 con 0x55cfdd40b840 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f10177fe640 1 -- 172.21.15.16:0/2105738138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10280043f0 con 0x55cfdd40b840 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.884+0000 7f10177fe640 1 -- 172.21.15.16:0/2105738138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102802d3f0 con 0x55cfdd40b840 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- 172.21.15.16:0/2105738138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 msgr2=0x55cfdd7df0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1028009230 tx=0x7f1028005540 comp rx=0 tx=0).stop 2024-09-21T02:10:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- 172.21.15.16:0/2105738138 shutdown_connections 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd7f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd40b840 0x55cfdd7df0b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 --2- 172.21.15.16:0/2105738138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd403660 0x55cfdd7e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- 172.21.15.16:0/2105738138 >> 172.21.15.16:0/2105738138 conn(0x55cfdd420910 msgr2=0x55cfdd7dcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- 172.21.15.16:0/2105738138 shutdown_connections 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- 172.21.15.16:0/2105738138 wait complete. 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.885+0000 7f103a718740 1 -- start start 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 Processor -- start 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd853ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b44e0 con 0x55cfdd403660 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b3e10 con 0x55cfdd42c520 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f102c1b1050 con 0x55cfdd40b840 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103a718740 10 client.0 _ll_register_callbacks cb 0x7f101c031400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103173f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103173f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7e4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58718/0 (socket says 172.21.15.16:58718) 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f103173f640 1 -- 172.21.15.16:0/3529142051 learned_addr learned my addr 172.21.15.16:0/3529142051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.886+0000 7f1031f40640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 -- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 msgr2=0x55cfdd853ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd853ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 -- 172.21.15.16:0/3529142051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 msgr2=0x55cfdd7e4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7e4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 -- 172.21.15.16:0/3529142051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfdd854120 con 0x55cfdd40b840 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f1031f40640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7e4fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1028011850 tx=0x7f10280096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=301f0f5a4dd67452 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f10157fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.887+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102803f060 con 0x55cfdd40b840 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10280043f0 con 0x55cfdd40b840 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f10280035f0 con 0x55cfdd40b840 2024-09-21T02:10:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1028046050 con 0x55cfdd40b840 2024-09-21T02:10:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 10 client.25597 fetch_fsmap learned FSMap version 28 2024-09-21T02:10:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cfdd85b4c0 con 0x55cfdd40b840 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10280043f0 con 0x55cfdd40b840 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f102802d580 con 0x55cfdd40b840 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 10 client.25597 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 10 client.25597 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 20 client.25597 populate_metadata read hostname 'smithi016' 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3c0250 0x55cfdd85da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 4 client.25597 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2a4c270d", "release": true, "await": true}] 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.888+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2a4c270d", "release": true, "await": true}) -- 0x55cfdd747bf0 con 0x55cfdd3c0250 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.889+0000 7f103173f640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3c0250 0x55cfdd85da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.889+0000 7f10157fa640 10 client.25597 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.889+0000 7f103173f640 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3c0250 0x55cfdd85da30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fe584a0eb377e61 server_cookie=b2deef13c8cb1c8 in_seq=0 out_seq=0 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f10157fa640 1 -- 172.21.15.16:0/3529142051 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cfdd747bf0 con 0x55cfdd3c0250 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f10157fa640 10 client.25597 handle_command_reply: tid=0 2024-09-21T02:10:26.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 client.25597 shutdown 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 msgr2=0x55cfdd7e4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7e4fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1028011850 tx=0x7f10280096d0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 shutdown_connections 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cfdd3c0250 0x55cfdd85da30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cfdd42c520 0x55cfdd853ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f10157fa640 0 client.25597 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfdd40b840 0x55cfdd7e4fe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 --2- 172.21.15.16:0/3529142051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cfdd403660 0x55cfdd7e4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.895+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 >> 172.21.15.16:0/3529142051 conn(0x55cfdd420910 msgr2=0x55cfdd7dcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.896+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 shutdown_connections 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.896+0000 7f103a718740 1 -- 172.21.15.16:0/3529142051 wait complete. 2024-09-21T02:10:26.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.896+0000 7f103a718740 20 client.25597 trim_cache size 0 max 16384 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 16, 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "2a4c270d": { 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 16, 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:10:26.909 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 147.8, 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:10:26.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:10:26.911 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:10:26.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 msgr2=0x7f10180b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 0x7f10180b8b90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f10240e59a0 tx=0x7f10240aa1d0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 msgr2=0x7f102c076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f102400eb10 tx=0x7f102400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 shutdown_connections 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f10180b6680 0x7f10180b8b90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f102c109df0 0x7f102c076190 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f102c1094a0 0x7f102c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 --2- 172.21.15.16:0/4218074921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f102c103b60 0x7f102c075710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 >> 172.21.15.16:0/4218074921 conn(0x7f102c07b850 msgr2=0x7f102c107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 shutdown_connections 2024-09-21T02:10:26.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:10:26.897+0000 7f10341a4640 1 -- 172.21.15.16:0/4218074921 wait complete. 2024-09-21T02:10:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:27 smithi016 ceph-mon[31245]: pgmap v451: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:27.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:27 smithi161 ceph-mon[34061]: pgmap v451: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:27.765 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2a4c270d', total seconds elapsed: 31.5 2024-09-21T02:10:27.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:27 smithi123 ceph-mon[33453]: pgmap v451: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:10:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:29 smithi016 ceph-mon[31245]: pgmap v452: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 581 KiB/s wr, 9 op/s 2024-09-21T02:10:29.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:29 smithi161 ceph-mon[34061]: pgmap v452: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 581 KiB/s wr, 9 op/s 2024-09-21T02:10:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:29 smithi123 ceph-mon[33453]: pgmap v452: 129 pgs: 129 active+clean; 7.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 581 KiB/s wr, 9 op/s 2024-09-21T02:10:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:31 smithi016 ceph-mon[31245]: pgmap v453: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 14 MiB/s wr, 17 op/s 2024-09-21T02:10:31.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:31 smithi161 ceph-mon[34061]: pgmap v453: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 14 MiB/s wr, 17 op/s 2024-09-21T02:10:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:31 smithi123 ceph-mon[33453]: pgmap v453: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 14 MiB/s wr, 17 op/s 2024-09-21T02:10:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:33 smithi016 ceph-mon[31245]: pgmap v454: 129 pgs: 129 active+clean; 7.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 35 MiB/s wr, 27 op/s 2024-09-21T02:10:33.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:33 smithi161 ceph-mon[34061]: pgmap v454: 129 pgs: 129 active+clean; 7.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 35 MiB/s wr, 27 op/s 2024-09-21T02:10:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:33 smithi123 ceph-mon[33453]: pgmap v454: 129 pgs: 129 active+clean; 7.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 35 MiB/s wr, 27 op/s 2024-09-21T02:10:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:35 smithi016 ceph-mon[31245]: pgmap v455: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 69 MiB/s wr, 46 op/s 2024-09-21T02:10:35.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:35 smithi161 ceph-mon[34061]: pgmap v455: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 69 MiB/s wr, 46 op/s 2024-09-21T02:10:35.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:35 smithi123 ceph-mon[33453]: pgmap v455: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 69 MiB/s wr, 46 op/s 2024-09-21T02:10:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:37 smithi016 ceph-mon[31245]: pgmap v456: 129 pgs: 129 active+clean; 8.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 86 MiB/s wr, 52 op/s 2024-09-21T02:10:37.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:37 smithi161 ceph-mon[34061]: pgmap v456: 129 pgs: 129 active+clean; 8.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 86 MiB/s wr, 52 op/s 2024-09-21T02:10:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:37 smithi123 ceph-mon[33453]: pgmap v456: 129 pgs: 129 active+clean; 8.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 86 MiB/s wr, 52 op/s 2024-09-21T02:10:39.316 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:39 smithi016 ceph-mon[31245]: pgmap v457: 129 pgs: 129 active+clean; 8.5 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 106 MiB/s wr, 62 op/s 2024-09-21T02:10:39.572 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:39 smithi123 ceph-mon[33453]: pgmap v457: 129 pgs: 129 active+clean; 8.5 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 106 MiB/s wr, 62 op/s 2024-09-21T02:10:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:39 smithi161 ceph-mon[34061]: pgmap v457: 129 pgs: 129 active+clean; 8.5 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 106 MiB/s wr, 62 op/s 2024-09-21T02:10:40.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:40 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:10:40.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:40 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:10:40.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:40 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:10:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:41 smithi016 ceph-mon[31245]: pgmap v458: 129 pgs: 129 active+clean; 8.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 134 MiB/s wr, 78 op/s 2024-09-21T02:10:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:10:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:10:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:10:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:41 smithi161 ceph-mon[34061]: pgmap v458: 129 pgs: 129 active+clean; 8.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 134 MiB/s wr, 78 op/s 2024-09-21T02:10:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:10:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:10:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:10:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:41 smithi123 ceph-mon[33453]: pgmap v458: 129 pgs: 129 active+clean; 8.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 134 MiB/s wr, 78 op/s 2024-09-21T02:10:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:10:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:10:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:10:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:43 smithi016 ceph-mon[31245]: pgmap v459: 129 pgs: 129 active+clean; 9.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 148 MiB/s wr, 84 op/s 2024-09-21T02:10:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:43 smithi161 ceph-mon[34061]: pgmap v459: 129 pgs: 129 active+clean; 9.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 148 MiB/s wr, 84 op/s 2024-09-21T02:10:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:43 smithi123 ceph-mon[33453]: pgmap v459: 129 pgs: 129 active+clean; 9.2 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 148 MiB/s wr, 84 op/s 2024-09-21T02:10:45.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:45 smithi016 ceph-mon[31245]: pgmap v460: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-09-21T02:10:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:45 smithi161 ceph-mon[34061]: pgmap v460: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-09-21T02:10:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:45 smithi123 ceph-mon[33453]: pgmap v460: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 162 MiB/s wr, 92 op/s 2024-09-21T02:10:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:47 smithi016 ceph-mon[31245]: pgmap v461: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 145 MiB/s wr, 91 op/s 2024-09-21T02:10:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:47 smithi161 ceph-mon[34061]: pgmap v461: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 145 MiB/s wr, 91 op/s 2024-09-21T02:10:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:47 smithi123 ceph-mon[33453]: pgmap v461: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 145 MiB/s wr, 91 op/s 2024-09-21T02:10:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:49 smithi016 ceph-mon[31245]: pgmap v462: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 143 MiB/s wr, 122 op/s 2024-09-21T02:10:49.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:49 smithi161 ceph-mon[34061]: pgmap v462: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 143 MiB/s wr, 122 op/s 2024-09-21T02:10:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:49 smithi123 ceph-mon[33453]: pgmap v462: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 143 MiB/s wr, 122 op/s 2024-09-21T02:10:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:51 smithi016 ceph-mon[31245]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 138 MiB/s wr, 242 op/s 2024-09-21T02:10:51.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:51 smithi161 ceph-mon[34061]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 138 MiB/s wr, 242 op/s 2024-09-21T02:10:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:51 smithi123 ceph-mon[33453]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 138 MiB/s wr, 242 op/s 2024-09-21T02:10:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:53 smithi016 ceph-mon[31245]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 119 MiB/s wr, 371 op/s 2024-09-21T02:10:53.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:53 smithi161 ceph-mon[34061]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 119 MiB/s wr, 371 op/s 2024-09-21T02:10:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:53 smithi123 ceph-mon[33453]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 119 MiB/s wr, 371 op/s 2024-09-21T02:10:55.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:55 smithi016 ceph-mon[31245]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 105 MiB/s wr, 575 op/s 2024-09-21T02:10:55.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:55 smithi161 ceph-mon[34061]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 105 MiB/s wr, 575 op/s 2024-09-21T02:10:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:55 smithi123 ceph-mon[33453]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 105 MiB/s wr, 575 op/s 2024-09-21T02:10:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:57 smithi016 ceph-mon[31245]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 77 MiB/s wr, 652 op/s 2024-09-21T02:10:57.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:57 smithi161 ceph-mon[34061]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 77 MiB/s wr, 652 op/s 2024-09-21T02:10:57.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:57 smithi123 ceph-mon[33453]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 77 MiB/s wr, 652 op/s 2024-09-21T02:10:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:10:59 smithi016 ceph-mon[31245]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 66 MiB/s wr, 737 op/s 2024-09-21T02:10:59.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:10:59 smithi161 ceph-mon[34061]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 66 MiB/s wr, 737 op/s 2024-09-21T02:10:59.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:10:59 smithi123 ceph-mon[33453]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 66 MiB/s wr, 737 op/s 2024-09-21T02:11:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:01 smithi016 ceph-mon[31245]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 61 MiB/s wr, 875 op/s 2024-09-21T02:11:01.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:01 smithi161 ceph-mon[34061]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 61 MiB/s wr, 875 op/s 2024-09-21T02:11:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:01 smithi123 ceph-mon[33453]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 61 MiB/s wr, 875 op/s 2024-09-21T02:11:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:03 smithi016 ceph-mon[31245]: pgmap v469: 129 pgs: 129 active+clean; 11 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 55 MiB/s wr, 895 op/s 2024-09-21T02:11:03.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:03 smithi161 ceph-mon[34061]: pgmap v469: 129 pgs: 129 active+clean; 11 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 55 MiB/s wr, 895 op/s 2024-09-21T02:11:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:03 smithi123 ceph-mon[33453]: pgmap v469: 129 pgs: 129 active+clean; 11 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 55 MiB/s wr, 895 op/s 2024-09-21T02:11:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:05 smithi016 ceph-mon[31245]: pgmap v470: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 976 op/s 2024-09-21T02:11:05.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:05 smithi161 ceph-mon[34061]: pgmap v470: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 976 op/s 2024-09-21T02:11:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:05 smithi123 ceph-mon[33453]: pgmap v470: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 976 op/s 2024-09-21T02:11:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:07 smithi016 ceph-mon[31245]: pgmap v471: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 52 MiB/s wr, 849 op/s 2024-09-21T02:11:07.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:07 smithi161 ceph-mon[34061]: pgmap v471: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 52 MiB/s wr, 849 op/s 2024-09-21T02:11:07.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:07 smithi123 ceph-mon[33453]: pgmap v471: 129 pgs: 129 active+clean; 11 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 52 MiB/s wr, 849 op/s 2024-09-21T02:11:09.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:09 smithi016 ceph-mon[31245]: pgmap v472: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 MiB/s wr, 843 op/s 2024-09-21T02:11:09.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:09 smithi161 ceph-mon[34061]: pgmap v472: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 MiB/s wr, 843 op/s 2024-09-21T02:11:09.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:09 smithi123 ceph-mon[33453]: pgmap v472: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 51 MiB/s wr, 843 op/s 2024-09-21T02:11:11.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:11 smithi016 ceph-mon[31245]: pgmap v473: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 914 op/s 2024-09-21T02:11:11.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:11 smithi161 ceph-mon[34061]: pgmap v473: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 914 op/s 2024-09-21T02:11:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:11 smithi123 ceph-mon[33453]: pgmap v473: 129 pgs: 129 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 914 op/s 2024-09-21T02:11:13.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:13 smithi016 ceph-mon[31245]: pgmap v474: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 MiB/s wr, 889 op/s 2024-09-21T02:11:13.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:13 smithi161 ceph-mon[34061]: pgmap v474: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 MiB/s wr, 889 op/s 2024-09-21T02:11:13.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:13 smithi123 ceph-mon[33453]: pgmap v474: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 54 MiB/s wr, 889 op/s 2024-09-21T02:11:15.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:15 smithi016 ceph-mon[31245]: pgmap v475: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 MiB/s wr, 954 op/s 2024-09-21T02:11:15.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:15 smithi161 ceph-mon[34061]: pgmap v475: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 MiB/s wr, 954 op/s 2024-09-21T02:11:15.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:15 smithi123 ceph-mon[33453]: pgmap v475: 129 pgs: 129 active+clean; 11 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 58 MiB/s wr, 954 op/s 2024-09-21T02:11:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:17 smithi016 ceph-mon[31245]: pgmap v476: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 823 op/s 2024-09-21T02:11:17.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:17 smithi161 ceph-mon[34061]: pgmap v476: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 823 op/s 2024-09-21T02:11:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:17 smithi123 ceph-mon[33453]: pgmap v476: 129 pgs: 129 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 823 op/s 2024-09-21T02:11:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:19 smithi016 ceph-mon[31245]: pgmap v477: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 827 op/s 2024-09-21T02:11:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:19 smithi161 ceph-mon[34061]: pgmap v477: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 827 op/s 2024-09-21T02:11:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:19 smithi123 ceph-mon[33453]: pgmap v477: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 51 MiB/s wr, 827 op/s 2024-09-21T02:11:21.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:21 smithi016 ceph-mon[31245]: pgmap v478: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 56 MiB/s wr, 906 op/s 2024-09-21T02:11:21.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:21 smithi161 ceph-mon[34061]: pgmap v478: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 56 MiB/s wr, 906 op/s 2024-09-21T02:11:21.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:21 smithi123 ceph-mon[33453]: pgmap v478: 129 pgs: 129 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 56 MiB/s wr, 906 op/s 2024-09-21T02:11:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:23 smithi016 ceph-mon[31245]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:23.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:23 smithi161 ceph-mon[34061]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:23.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:23 smithi123 ceph-mon[33453]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:25 smithi016 ceph-mon[31245]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-09-21T02:11:25.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:25 smithi161 ceph-mon[34061]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-09-21T02:11:25.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:25 smithi123 ceph-mon[33453]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 58 MiB/s wr, 935 op/s 2024-09-21T02:11:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:27 smithi016 ceph-mon[31245]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:11:27.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:27 smithi161 ceph-mon[34061]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:11:27.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:27 smithi123 ceph-mon[33453]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:11:29.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:29 smithi161 ceph-mon[34061]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:29 smithi123 ceph-mon[33453]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:29.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:29 smithi016 ceph-mon[31245]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:31.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:31 smithi161 ceph-mon[34061]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 54 MiB/s wr, 876 op/s 2024-09-21T02:11:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:31 smithi123 ceph-mon[33453]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 54 MiB/s wr, 876 op/s 2024-09-21T02:11:31.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:31 smithi016 ceph-mon[31245]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 54 MiB/s wr, 876 op/s 2024-09-21T02:11:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:33 smithi123 ceph-mon[33453]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:33.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:33 smithi016 ceph-mon[31245]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:33.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:33 smithi161 ceph-mon[34061]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:35.716 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:35 smithi161 ceph-mon[34061]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-09-21T02:11:35.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:35 smithi123 ceph-mon[33453]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-09-21T02:11:35.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:35 smithi016 ceph-mon[31245]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-09-21T02:11:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:37 smithi123 ceph-mon[33453]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 49 MiB/s wr, 799 op/s 2024-09-21T02:11:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:37 smithi016 ceph-mon[31245]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 49 MiB/s wr, 799 op/s 2024-09-21T02:11:37.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:37 smithi161 ceph-mon[34061]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 49 MiB/s wr, 799 op/s 2024-09-21T02:11:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:39 smithi123 ceph-mon[33453]: pgmap v487: 129 pgs: 129 active+clean; 12 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:39 smithi016 ceph-mon[31245]: pgmap v487: 129 pgs: 129 active+clean; 12 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:39.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:39 smithi161 ceph-mon[34061]: pgmap v487: 129 pgs: 129 active+clean; 12 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 50 MiB/s wr, 806 op/s 2024-09-21T02:11:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:41 smithi123 ceph-mon[33453]: pgmap v488: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 54 MiB/s wr, 878 op/s 2024-09-21T02:11:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:11:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:11:41.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:11:41.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:41 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:11:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:41 smithi016 ceph-mon[31245]: pgmap v488: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 54 MiB/s wr, 878 op/s 2024-09-21T02:11:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:11:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:11:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:11:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:41 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:11:41.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:41 smithi161 ceph-mon[34061]: pgmap v488: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 54 MiB/s wr, 878 op/s 2024-09-21T02:11:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:11:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:11:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:11:41.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:41 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:11:43.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:43 smithi016 ceph-mon[31245]: pgmap v489: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:43 smithi123 ceph-mon[33453]: pgmap v489: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:43.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:43 smithi161 ceph-mon[34061]: pgmap v489: 129 pgs: 129 active+clean; 12 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 53 MiB/s wr, 852 op/s 2024-09-21T02:11:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:45 smithi123 ceph-mon[33453]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 904 op/s 2024-09-21T02:11:45.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:45 smithi016 ceph-mon[31245]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 904 op/s 2024-09-21T02:11:45.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:45 smithi161 ceph-mon[34061]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 904 op/s 2024-09-21T02:11:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:47 smithi123 ceph-mon[33453]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 781 op/s 2024-09-21T02:11:47.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:47 smithi016 ceph-mon[31245]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 781 op/s 2024-09-21T02:11:47.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:47 smithi161 ceph-mon[34061]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 781 op/s 2024-09-21T02:11:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:49 smithi123 ceph-mon[33453]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 783 op/s 2024-09-21T02:11:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:49 smithi016 ceph-mon[31245]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 783 op/s 2024-09-21T02:11:49.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:49 smithi161 ceph-mon[34061]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 48 MiB/s wr, 783 op/s 2024-09-21T02:11:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:51 smithi123 ceph-mon[33453]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 MiB/s wr, 847 op/s 2024-09-21T02:11:51.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:51 smithi016 ceph-mon[31245]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 MiB/s wr, 847 op/s 2024-09-21T02:11:51.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:51 smithi161 ceph-mon[34061]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 52 MiB/s wr, 847 op/s 2024-09-21T02:11:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:53 smithi123 ceph-mon[33453]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-21T02:11:53.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:53 smithi016 ceph-mon[31245]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-21T02:11:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:53 smithi161 ceph-mon[34061]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-21T02:11:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:55 smithi123 ceph-mon[33453]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:55 smithi016 ceph-mon[31245]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:55.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:55 smithi161 ceph-mon[34061]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 54 MiB/s wr, 877 op/s 2024-09-21T02:11:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:57 smithi123 ceph-mon[33453]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 766 op/s 2024-09-21T02:11:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:57 smithi016 ceph-mon[31245]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 766 op/s 2024-09-21T02:11:57.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:57 smithi161 ceph-mon[34061]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 766 op/s 2024-09-21T02:11:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:11:59 smithi123 ceph-mon[33453]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 767 op/s 2024-09-21T02:11:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:11:59 smithi016 ceph-mon[31245]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 767 op/s 2024-09-21T02:11:59.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:11:59 smithi161 ceph-mon[34061]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 47 MiB/s wr, 767 op/s 2024-09-21T02:12:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:01 smithi016 ceph-mon[31245]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 51 MiB/s wr, 832 op/s 2024-09-21T02:12:02.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:01 smithi161 ceph-mon[34061]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 51 MiB/s wr, 832 op/s 2024-09-21T02:12:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:01 smithi123 ceph-mon[33453]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 51 MiB/s wr, 832 op/s 2024-09-21T02:12:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:03 smithi016 ceph-mon[31245]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:12:04.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:03 smithi161 ceph-mon[34061]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:12:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:03 smithi123 ceph-mon[33453]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 50 MiB/s wr, 809 op/s 2024-09-21T02:12:06.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:05 smithi016 ceph-mon[31245]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 872 op/s 2024-09-21T02:12:06.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:05 smithi161 ceph-mon[34061]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 872 op/s 2024-09-21T02:12:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:05 smithi123 ceph-mon[33453]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 872 op/s 2024-09-21T02:12:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:07 smithi016 ceph-mon[31245]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 749 op/s 2024-09-21T02:12:08.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:07 smithi161 ceph-mon[34061]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 749 op/s 2024-09-21T02:12:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:07 smithi123 ceph-mon[33453]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 749 op/s 2024-09-21T02:12:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:09 smithi016 ceph-mon[31245]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 743 op/s 2024-09-21T02:12:10.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:09 smithi161 ceph-mon[34061]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 743 op/s 2024-09-21T02:12:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:09 smithi123 ceph-mon[33453]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 MiB/s wr, 743 op/s 2024-09-21T02:12:11.024 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-09-21T02:12:11.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:12:11.571 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:11 smithi016 ceph-mon[31245]: pgmap v503: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 48 MiB/s wr, 798 op/s 2024-09-21T02:12:12.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:11 smithi161 ceph-mon[34061]: pgmap v503: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 48 MiB/s wr, 798 op/s 2024-09-21T02:12:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:11 smithi123 ceph-mon[33453]: pgmap v503: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 48 MiB/s wr, 798 op/s 2024-09-21T02:12:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:13 smithi016 ceph-mon[31245]: pgmap v504: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 47 MiB/s wr, 769 op/s 2024-09-21T02:12:14.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:13 smithi161 ceph-mon[34061]: pgmap v504: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 47 MiB/s wr, 769 op/s 2024-09-21T02:12:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:13 smithi123 ceph-mon[33453]: pgmap v504: 129 pgs: 129 active+clean; 14 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 47 MiB/s wr, 769 op/s 2024-09-21T02:12:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:15 smithi016 ceph-mon[31245]: pgmap v505: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 49 MiB/s wr, 809 op/s 2024-09-21T02:12:16.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:15 smithi161 ceph-mon[34061]: pgmap v505: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 49 MiB/s wr, 809 op/s 2024-09-21T02:12:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:15 smithi123 ceph-mon[33453]: pgmap v505: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 49 MiB/s wr, 809 op/s 2024-09-21T02:12:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:17 smithi016 ceph-mon[31245]: pgmap v506: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:12:18.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:17 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2413426671' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:17 smithi161 ceph-mon[34061]: pgmap v506: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:12:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:17 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2413426671' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:17 smithi123 ceph-mon[33453]: pgmap v506: 129 pgs: 129 active+clean; 14 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:12:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:17 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2413426671' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 -- 172.21.15.16:0/1707948053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c102f50 msgr2=0x7f620c10fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 --2- 172.21.15.16:0/1707948053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c102f50 0x7f620c10fc70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6200009900 tx=0x7f620002a480 comp rx=0 tx=0).stop 2024-09-21T02:12:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 -- 172.21.15.16:0/1707948053 shutdown_connections 2024-09-21T02:12:18.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 --2- 172.21.15.16:0/1707948053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c1101b0 0x7f620c1125d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 --2- 172.21.15.16:0/1707948053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c102f50 0x7f620c10fc70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 --2- 172.21.15.16:0/1707948053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102600 0x7f620c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 -- 172.21.15.16:0/1707948053 >> 172.21.15.16:0/1707948053 conn(0x7f620c079750 msgr2=0x7f620c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 -- 172.21.15.16:0/1707948053 shutdown_connections 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.916+0000 7f6210ccd640 1 -- 172.21.15.16:0/1707948053 wait complete. 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 Processor -- start 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 -- start start 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 0x7f620c19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102f50 0x7f620c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f620c115b50 con 0x7f620c1101b0 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f620c1159d0 con 0x7f620c102600 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6210ccd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f620c115cd0 con 0x7f620c102f50 2024-09-21T02:12:18.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f620ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f620ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53626/0 (socket says 172.21.15.16:53626) 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f620ad9d640 1 -- 172.21.15.16:0/2413426671 learned_addr learned my addr 172.21.15.16:0/2413426671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f6209d9b640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102f50 0x7f620c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.917+0000 7f620a59c640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 0x7f620c19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 -- 172.21.15.16:0/2413426671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102f50 msgr2=0x7f620c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102f50 0x7f620c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 -- 172.21.15.16:0/2413426671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 msgr2=0x7f620c19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 0x7f620c19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 -- 172.21.15.16:0/2413426671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c1a2510 con 0x7f620c1101b0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620a59c640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 0x7f620c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f620ad9d640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f620400eb10 tx=0x7f620400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204004b20 con 0x7f620c1101b0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6204004cc0 con 0x7f620c1101b0 2024-09-21T02:12:18.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c1a2800 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204010690 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.918+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c1a8f60 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.919+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c06c280 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.921+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6204005000 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.922+0000 7f61fb7fe640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 0x7f61f40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.922+0000 7f620a59c640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 0x7f61f40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.922+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f62040e81f0 con 0x7f620c1101b0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.923+0000 7f620a59c640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 0x7f61f40b8b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009fc0 tx=0x7f61fc005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:12:18.518 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:16.926+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6204056710 con 0x7f620c1101b0 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.224+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f620c102a10 con 0x7f620c1101b0 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.226+0000 7f61fb7fe640 1 -- 172.21.15.16:0/2413426671 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f620c102a10 con 0x7f620c1101b0 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.233+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 msgr2=0x7f61f40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.233+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 0x7f61f40b8b90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009fc0 tx=0x7f61fc005f90 comp rx=0 tx=0).stop 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.233+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 msgr2=0x7f620c1a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.233+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f620400eb10 tx=0x7f620400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 shutdown_connections 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f61f40b6680 0x7f61f40b8b90 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009fc0 tx=0x7f61fc005f90 comp rx=0 tx=0).stop 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620c1101b0 0x7f620c1a1de0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f620c102f50 0x7f620c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 --2- 172.21.15.16:0/2413426671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f620c102600 0x7f620c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 >> 172.21.15.16:0/2413426671 conn(0x7f620c079750 msgr2=0x7f620c111670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.234+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 shutdown_connections 2024-09-21T02:12:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:17.235+0000 7f6210ccd640 1 -- 172.21.15.16:0/2413426671 wait complete. 2024-09-21T02:12:20.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:19 smithi016 ceph-mon[31245]: pgmap v507: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:12:20.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:19 smithi161 ceph-mon[34061]: pgmap v507: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:12:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:19 smithi123 ceph-mon[33453]: pgmap v507: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:12:22.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:21 smithi016 ceph-mon[31245]: pgmap v508: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 689 op/s 2024-09-21T02:12:22.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:21 smithi161 ceph-mon[34061]: pgmap v508: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 689 op/s 2024-09-21T02:12:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:21 smithi123 ceph-mon[33453]: pgmap v508: 129 pgs: 129 active+clean; 14 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 689 op/s 2024-09-21T02:12:23.028 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-09-21T02:12:23.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-09-21T02:12:23.784 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:23 smithi016 ceph-mon[31245]: pgmap v509: 129 pgs: 129 active+clean; 14 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 40 MiB/s wr, 642 op/s 2024-09-21T02:12:24.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:23 smithi161 ceph-mon[34061]: pgmap v509: 129 pgs: 129 active+clean; 14 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 40 MiB/s wr, 642 op/s 2024-09-21T02:12:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:23 smithi123 ceph-mon[33453]: pgmap v509: 129 pgs: 129 active+clean; 14 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 40 MiB/s wr, 642 op/s 2024-09-21T02:12:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:24 smithi123 ceph-mon[33453]: pgmap v510: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 41 MiB/s wr, 662 op/s 2024-09-21T02:12:25.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:24 smithi016 ceph-mon[31245]: pgmap v510: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 41 MiB/s wr, 662 op/s 2024-09-21T02:12:25.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:24 smithi161 ceph-mon[34061]: pgmap v510: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 41 MiB/s wr, 662 op/s 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.493+0000 7f2f177fe640 1 -- 172.21.15.16:0/63565609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10030c40 con 0x7f2f18108b30 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.494+0000 7f2f1dba4640 1 -- 172.21.15.16:0/63565609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f18108b30 msgr2=0x7f2f1810f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.494+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/63565609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f18108b30 0x7f2f1810f360 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009be0 tx=0x7f2f1002a6f0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.495+0000 7f2f1dba4640 1 -- 172.21.15.16:0/63565609 shutdown_connections 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.495+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/63565609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f18108b30 0x7f2f1810f360 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.495+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/63565609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181081e0 0x7f2f181085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.495+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/63565609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f181028a0 0x7f2f18102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.495+0000 7f2f1dba4640 1 -- 172.21.15.16:0/63565609 >> 172.21.15.16:0/63565609 conn(0x7f2f180fe330 msgr2=0x7f2f18100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.497+0000 7f2f1dba4640 1 -- 172.21.15.16:0/63565609 shutdown_connections 2024-09-21T02:12:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.498+0000 7f2f1dba4640 1 -- 172.21.15.16:0/63565609 wait complete. 2024-09-21T02:12:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 Processor -- start 2024-09-21T02:12:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 -- start start 2024-09-21T02:12:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 0x7f2f1806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 0x7f2f1806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f18108b30 0x7f2f18072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181115c0 con 0x7f2f181081e0 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18111440 con 0x7f2f181028a0 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.499+0000 7f2f1dba4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18111740 con 0x7f2f18108b30 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.500+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 0x7f2f1806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.501+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 0x7f2f1806d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45468/0 (socket says 172.21.15.16:45468) 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.501+0000 7f2f17fff640 1 -- 172.21.15.16:0/3631270066 learned_addr learned my addr 172.21.15.16:0/3631270066 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.502+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f18108b30 0x7f2f18072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.502+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 0x7f2f1806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.503+0000 7f2f1cba2640 1 -- 172.21.15.16:0/3631270066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f18108b30 msgr2=0x7f2f18072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.503+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f18108b30 0x7f2f18072910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.503+0000 7f2f1cba2640 1 -- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 msgr2=0x7f2f1806d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.503+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 0x7f2f1806d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.503+0000 7f2f1cba2640 1 -- 172.21.15.16:0/3631270066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f18073040 con 0x7f2f181028a0 2024-09-21T02:12:26.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 0x7f2f1806d0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00d6d0 tx=0x7f2f0c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f15ffb640 1 -- 172.21.15.16:0/3631270066 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c017070 con 0x7f2f181028a0 2024-09-21T02:12:26.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f15ffb640 1 -- 172.21.15.16:0/3631270066 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f0c010460 con 0x7f2f181028a0 2024-09-21T02:12:26.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f180732d0 con 0x7f2f181028a0 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f15ffb640 1 -- 172.21.15.16:0/3631270066 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00f620 con 0x7f2f181028a0 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.505+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f18073a80 con 0x7f2f181028a0 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.510+0000 7f2f2537a740 1 Processor -- start 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 -- start start 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febc2980 0x5651fef92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febcab60 0x5651fef93240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651fef08b90 con 0x5651febcab60 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5651feef7fc0 con 0x5651febc2980 2024-09-21T02:12:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.511+0000 7f2f2537a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651feefbdd0 con 0x5651febeb840 2024-09-21T02:12:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.514+0000 7f2f1d3a3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.514+0000 7f2f1d3a3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:51800/0 (socket says 172.21.15.16:51800) 2024-09-21T02:12:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.514+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/2404677603 learned_addr learned my addr 172.21.15.16:0/2404677603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f15ffb640 1 -- 172.21.15.16:0/3631270066 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2f0c01a070 con 0x7f2f181028a0 2024-09-21T02:12:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/2404677603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febc2980 msgr2=0x5651fef92d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febc2980 0x5651fef92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/2404677603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febcab60 msgr2=0x5651fef93240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febcab60 0x5651fef93240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/2404677603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651fee7f060 con 0x5651febeb840 2024-09-21T02:12:26.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.515+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006010 tx=0x7f2f10009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a7ff158017d1e74 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.518+0000 7f2f15ffb640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 0x7f2f080b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.518+0000 7f2f15ffb640 1 -- 172.21.15.16:0/3631270066 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2f0c0e6d50 con 0x7f2f181028a0 2024-09-21T02:12:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.519+0000 7f2f02ffd640 1 -- 172.21.15.16:0/2404677603 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10040030 con 0x5651febeb840 2024-09-21T02:12:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.519+0000 7f2f02ffd640 1 -- 172.21.15.16:0/2404677603 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f10037040 con 0x5651febeb840 2024-09-21T02:12:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.520+0000 7f2f2537a740 1 -- 172.21.15.16:0/2404677603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 msgr2=0x5651fef927c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.520+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006010 tx=0x7f2f10009370 comp rx=0 tx=0).stop 2024-09-21T02:12:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.520+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 0x7f2f080b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.521+0000 7f2f1cba2640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 0x7f2f080b8c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5651fef91440 tx=0x7f2f0c010f60 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 -- 172.21.15.16:0/2404677603 shutdown_connections 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febcab60 0x5651fef93240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febc2980 0x5651fef92d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2404677603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fef927c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 -- 172.21.15.16:0/2404677603 >> 172.21.15.16:0/2404677603 conn(0x5651febdf990 msgr2=0x5651fef906b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 -- 172.21.15.16:0/2404677603 shutdown_connections 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.522+0000 7f2f2537a740 1 -- 172.21.15.16:0/2404677603 wait complete. 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.523+0000 7f2f2537a740 1 -- start start 2024-09-21T02:12:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.523+0000 7f2f2537a740 1 Processor -- start 2024-09-21T02:12:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.528+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef9e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febeb840 0x5651fefaa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f2537a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5651fed597a0 con 0x5651febc2980 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:51804/0 (socket says 172.21.15.16:51804) 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f17fff640 1 -- 172.21.15.16:0/2058332332 learned_addr learned my addr 172.21.15.16:0/2058332332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5651fee9d0a0 con 0x5651febeb840 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.529+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5651fefa0f90 con 0x5651febcab60 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f2537a740 10 client.0 _ll_register_callbacks cb 0x5651fee7a3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 -- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febeb840 msgr2=0x5651fefaa2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:12:26.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febeb840 0x5651fefaa2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 -- 172.21.15.16:0/2058332332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 msgr2=0x5651fef9e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef9e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651fee8b880 con 0x5651febcab60 2024-09-21T02:12:26.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f17fff640 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f0402d6d0 tx=0x7f2f040327b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e7c25f1d7503c10 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.530+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f10037040 con 0x5651febcab60 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2ef7fff640 10 client.25615 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2ef7fff640 1 -- 172.21.15.16:0/2058332332 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04033860 con 0x5651febcab60 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2ef7fff640 1 -- 172.21.15.16:0/2058332332 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2f04033a00 con 0x5651febcab60 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2ef7fff640 1 -- 172.21.15.16:0/2058332332 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f04033c20 con 0x5651febcab60 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2f2537a740 10 client.25615 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.531+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5651fec65b90 con 0x5651febcab60 2024-09-21T02:12:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.532+0000 7f2ef7fff640 1 -- 172.21.15.16:0/2058332332 <== mon.1 v2:172.21.15.161:3300/0 4 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f2f04030040 con 0x5651febcab60 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 10 client.25615 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 10 client.25615 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 20 client.25615 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651febaa1c0 0x5651fefb0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 4 client.25615 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2f10037040 con 0x5651febaa1c0 2024-09-21T02:12:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.533+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651febaa1c0 0x5651fefb0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.536+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651febaa1c0 0x5651fefb0740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c3ce9d25542acf7 server_cookie=3d8ecaad5cfe2118 in_seq=0 out_seq=0 2024-09-21T02:12:26.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.536+0000 7f2ef7fff640 10 client.25615 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.539+0000 7f2ef7fff640 1 -- 172.21.15.16:0/2058332332 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x7f2f10037040 con 0x5651febaa1c0 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.539+0000 7f2ef7fff640 10 client.25615 handle_command_reply: tid=0 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 client.25615 shutdown 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 msgr2=0x5651fef9e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f0402d6d0 tx=0x7f2f040327b0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 shutdown_connections 2024-09-21T02:12:26.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651febaa1c0 0x5651fefb0740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febeb840 0x5651fefaa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febcab60 0x5651fef9e7f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 --2- 172.21.15.16:0/2058332332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef9e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 >> 172.21.15.16:0/2058332332 conn(0x5651febdf990 msgr2=0x5651fef8ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.540+0000 7f2ef7fff640 0 client.25615 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.544+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 shutdown_connections 2024-09-21T02:12:26.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.545+0000 7f2f2537a740 1 -- 172.21.15.16:0/2058332332 wait complete. 2024-09-21T02:12:26.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.545+0000 7f2f2537a740 20 client.25615 trim_cache size 0 max 16384 2024-09-21T02:12:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.580+0000 7f2f2537a740 1 Processor -- start 2024-09-21T02:12:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.580+0000 7f2f2537a740 1 -- start start 2024-09-21T02:12:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.581+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef905e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.581+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.581+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefb5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:52878/0 (socket says 172.21.15.16:52878) 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 -- 172.21.15.16:0/3149107086 learned_addr learned my addr 172.21.15.16:0/3149107086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef905e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.581+0000 7f2f2537a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b72f0 con 0x5651febc2980 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b8e30 con 0x5651febcab60 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b79c0 con 0x5651febeb840 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 -- 172.21.15.16:0/3149107086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 msgr2=0x5651fefb5f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefb5f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 -- 172.21.15.16:0/3149107086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 msgr2=0x5651fef905e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef905e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.582+0000 7f2f17fff640 1 -- 172.21.15.16:0/3149107086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651fef95620 con 0x5651febcab60 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.583+0000 7f2f17fff640 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f04005f60 tx=0x7f2f04035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5b5cb0c8d76deb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.583+0000 7f2f027fc640 1 -- 172.21.15.16:0/3149107086 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04033860 con 0x5651febcab60 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.583+0000 7f2f027fc640 1 -- 172.21.15.16:0/3149107086 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f04033a00 con 0x5651febcab60 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.583+0000 7f2f027fc640 1 -- 172.21.15.16:0/3149107086 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0403ecb0 con 0x5651febcab60 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.584+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 msgr2=0x5651fef93f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.584+0000 7f2f2537a740 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f04005f60 tx=0x7f2f04035000 comp rx=0 tx=0).stop 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 shutdown_connections 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefb5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651febcab60 0x5651fef93f00 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f04005f60 tx=0x7f2f04035000 comp rx=0 tx=0).stop 2024-09-21T02:12:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 --2- 172.21.15.16:0/3149107086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fef905e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 >> 172.21.15.16:0/3149107086 conn(0x5651febdf990 msgr2=0x5651fef937f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 shutdown_connections 2024-09-21T02:12:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.585+0000 7f2f2537a740 1 -- 172.21.15.16:0/3149107086 wait complete. 2024-09-21T02:12:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 -- start start 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 Processor -- start 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fefa11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefa1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b8760 con 0x5651febc2980 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b8090 con 0x5651feb7f570 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181b52d0 con 0x5651febeb840 2024-09-21T02:12:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.586+0000 7f2f2537a740 10 client.0 _ll_register_callbacks cb 0x7f2f0402d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:52882/0 (socket says 172.21.15.16:52882) 2024-09-21T02:12:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/1423495581 learned_addr learned my addr 172.21.15.16:0/1423495581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/1423495581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 msgr2=0x5651fefa1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefa1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/1423495581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 msgr2=0x5651fefa11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fefa11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.587+0000 7f2f1d3a3640 1 -- 172.21.15.16:0/1423495581 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651ff01b140 con 0x5651feb7f570 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.588+0000 7f2f1d3a3640 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002ac10 tx=0x7f2f10009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8792de9e58618a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.588+0000 7f2ef7fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.589+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10040040 con 0x5651feb7f570 2024-09-21T02:12:26.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.589+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f04033a00 con 0x5651feb7f570 2024-09-21T02:12:26.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.589+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f2f100305e0 con 0x5651feb7f570 2024-09-21T02:12:26.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.589+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f1005a050 con 0x5651feb7f570 2024-09-21T02:12:26.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.589+0000 7f2f2537a740 10 client.25535 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:26.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.590+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5651ff01c370 con 0x5651feb7f570 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.591+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2f04033a00 con 0x5651feb7f570 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f2f100043d0 con 0x5651feb7f570 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 10 client.25535 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 10 client.25535 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 20 client.25535 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651ff01cab0 0x5651ff01eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 4 client.25535 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-09-21T02:12:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x5651fef0aae0 con 0x5651ff01cab0 2024-09-21T02:12:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.592+0000 7f2f17fff640 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651ff01cab0 0x5651ff01eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.593+0000 7f2f17fff640 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651ff01cab0 0x5651ff01eed0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8da0ce8e3a30bbdd server_cookie=df77f92cfd3383b8 in_seq=0 out_seq=0 2024-09-21T02:12:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.593+0000 7f2ef7fff640 10 client.25535 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.649+0000 7f2ef7fff640 1 -- 172.21.15.16:0/1423495581 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5651fef0aae0 con 0x5651ff01cab0 2024-09-21T02:12:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.649+0000 7f2ef7fff640 10 client.25535 handle_command_reply: tid=0 2024-09-21T02:12:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.649+0000 7f2f2537a740 1 client.25535 shutdown 2024-09-21T02:12:26.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 msgr2=0x5651fefa1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f1002ac10 tx=0x7f2f10009ba0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 shutdown_connections 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5651ff01cab0 0x5651ff01eed0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5651feb7f570 0x5651fefa1c70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5651febeb840 0x5651fefa1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 --2- 172.21.15.16:0/1423495581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5651febc2980 0x5651fefa11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2ef7fff640 0 client.25535 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 >> 172.21.15.16:0/1423495581 conn(0x5651febdf990 msgr2=0x5651fef94120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 shutdown_connections 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 1 -- 172.21.15.16:0/1423495581 wait complete. 2024-09-21T02:12:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.650+0000 7f2f2537a740 20 client.25535 trim_cache size 0 max 16384 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 18, 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "ec7be942": { 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 18, 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:12:26.653 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:12:26.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:26.655 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 msgr2=0x7f2f080b8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 0x7f2f080b8c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5651fef91440 tx=0x7f2f0c010f60 comp rx=0 tx=0).stop 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 msgr2=0x7f2f1806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 0x7f2f1806d0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00d6d0 tx=0x7f2f0c00db90 comp rx=0 tx=0).stop 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 shutdown_connections 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f2f080b6770 0x7f2f080b8c80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f18108b30 0x7f2f18072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f181081e0 0x7f2f1806d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 --2- 172.21.15.16:0/3631270066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f181028a0 0x7f2f1806d0d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.651+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 >> 172.21.15.16:0/3631270066 conn(0x7f2f180fe330 msgr2=0x7f2f18100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.652+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 shutdown_connections 2024-09-21T02:12:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:26.652+0000 7f2f1dba4640 1 -- 172.21.15.16:0/3631270066 wait complete. 2024-09-21T02:12:27.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:27 smithi016 ceph-mon[31245]: pgmap v511: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 34 MiB/s wr, 559 op/s 2024-09-21T02:12:27.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:27 smithi123 ceph-mon[33453]: pgmap v511: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 34 MiB/s wr, 559 op/s 2024-09-21T02:12:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:27 smithi161 ceph-mon[34061]: pgmap v511: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 34 MiB/s wr, 559 op/s 2024-09-21T02:12:28.074 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ec7be942', quiesce took 17.0 seconds. Will release after: -7.0 2024-09-21T02:12:28.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:12:28.526 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 -- 172.21.15.16:0/3693802364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 msgr2=0x7effcc102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 --2- 172.21.15.16:0/3693802364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc102e70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7effbc009900 tx=0x7effbc02a3f0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 -- 172.21.15.16:0/3693802364 shutdown_connections 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 --2- 172.21.15.16:0/3693802364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc108cf0 0x7effcc10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 --2- 172.21.15.16:0/3693802364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc1083a0 0x7effcc1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 --2- 172.21.15.16:0/3693802364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc102e70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 -- 172.21.15.16:0/3693802364 >> 172.21.15.16:0/3693802364 conn(0x7effcc079730 msgr2=0x7effcc079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 -- 172.21.15.16:0/3693802364 shutdown_connections 2024-09-21T02:12:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.152+0000 7effd2619640 1 -- 172.21.15.16:0/3693802364 wait complete. 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 Processor -- start 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 -- start start 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 0x7effcc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 0x7effcc19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effcc113930 con 0x7effcc102a60 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7effcc1137b0 con 0x7effcc108cf0 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd2619640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7effcc113ab0 con 0x7effcc1083a0 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50878/0 (socket says 172.21.15.16:50878) 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effcbfff640 1 -- 172.21.15.16:0/2379797229 learned_addr learned my addr 172.21.15.16:0/2379797229 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effcb7fe640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 0x7effcc19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.153+0000 7effd0bb6640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 0x7effcc19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 -- 172.21.15.16:0/2379797229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 msgr2=0x7effcc19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 0x7effcc19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 -- 172.21.15.16:0/2379797229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 msgr2=0x7effcc19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 0x7effcc19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 -- 172.21.15.16:0/2379797229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effcc1a0190 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effd0bb6640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 0x7effcc19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcb7fe640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 0x7effcc19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effcbfff640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effbc009a20 tx=0x7effbc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc03f030 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effbc0035f0 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effcc1a0420 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc035070 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.154+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effcc1a0900 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.155+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effcc102e70 con 0x7effcc102a60 2024-09-21T02:12:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.157+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7effbc003aa0 con 0x7effcc102a60 2024-09-21T02:12:29.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.157+0000 7effc97fa640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 0x7effb40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:29.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.158+0000 7effcb7fe640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 0x7effb40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:29.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.158+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7effbc108710 con 0x7effcc102a60 2024-09-21T02:12:29.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.158+0000 7effcb7fe640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 0x7effb40b8bc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7effcc100b70 tx=0x7effc000a5c0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:29.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.162+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7effbc0ce890 con 0x7effcc102a60 2024-09-21T02:12:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.389+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effcc02e640 con 0x7effcc102a60 2024-09-21T02:12:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.389+0000 7effc97fa640 1 -- 172.21.15.16:0/2379797229 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7effbc0d2c70 con 0x7effcc102a60 2024-09-21T02:12:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:12:29.391 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:12:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:12:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.391+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 msgr2=0x7effb40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.391+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 0x7effb40b8bc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7effcc100b70 tx=0x7effc000a5c0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.391+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 msgr2=0x7effcc19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.391+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effbc009a20 tx=0x7effbc004270 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 shutdown_connections 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7effb40b66b0 0x7effb40b8bc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effcc108cf0 0x7effcc19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effcc1083a0 0x7effcc19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 --2- 172.21.15.16:0/2379797229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effcc102a60 0x7effcc19a2e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 >> 172.21.15.16:0/2379797229 conn(0x7effcc079730 msgr2=0x7effcc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 shutdown_connections 2024-09-21T02:12:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:29.392+0000 7effd2619640 1 -- 172.21.15.16:0/2379797229 wait complete. 2024-09-21T02:12:29.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:29 smithi123 ceph-mon[33453]: pgmap v512: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 35 MiB/s wr, 562 op/s 2024-09-21T02:12:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:29 smithi016 ceph-mon[31245]: pgmap v512: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 35 MiB/s wr, 562 op/s 2024-09-21T02:12:29.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:29 smithi161 ceph-mon[34061]: pgmap v512: 129 pgs: 129 active+clean; 14 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 35 MiB/s wr, 562 op/s 2024-09-21T02:12:30.213 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id ec7be942 --release --await' 2024-09-21T02:12:30.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id ec7be942 --release --await 2024-09-21T02:12:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:30.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:30 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2379797229' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:30.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:30 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2379797229' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:30.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:30 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2379797229' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 -- 172.21.15.16:0/3204399885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760103b60 msgr2=0x7ff760103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 --2- 172.21.15.16:0/3204399885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760103b60 0x7ff760103f70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009900 tx=0x7ff75c02a410 comp rx=0 tx=0).stop 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 -- 172.21.15.16:0/3204399885 shutdown_connections 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 --2- 172.21.15.16:0/3204399885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760109df0 0x7ff760110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 --2- 172.21.15.16:0/3204399885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 0x7ff7601098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 --2- 172.21.15.16:0/3204399885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760103b60 0x7ff760103f70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.962+0000 7ff7675eb640 1 -- 172.21.15.16:0/3204399885 >> 172.21.15.16:0/3204399885 conn(0x7ff76007b850 msgr2=0x7ff76007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.963+0000 7ff7675eb640 1 -- 172.21.15.16:0/3204399885 shutdown_connections 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.963+0000 7ff7675eb640 1 -- 172.21.15.16:0/3204399885 wait complete. 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.963+0000 7ff7675eb640 1 Processor -- start 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.963+0000 7ff7675eb640 1 -- start start 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760103b60 0x7ff76019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 0x7ff76019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff760113960 con 0x7ff760109df0 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601137e0 con 0x7ff7601094a0 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7675eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff760113ae0 con 0x7ff760103b60 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50896/0 (socket says 172.21.15.16:50896) 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 -- 172.21.15.16:0/553868602 learned_addr learned my addr 172.21.15.16:0/553868602 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff764b86640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 0x7ff76019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765387640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760103b60 0x7ff76019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 -- 172.21.15.16:0/553868602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760103b60 msgr2=0x7ff76019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760103b60 0x7ff76019c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 -- 172.21.15.16:0/553868602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 msgr2=0x7ff76019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 0x7ff76019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 -- 172.21.15.16:0/553868602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7601a2500 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff765b88640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff75800eaa0 tx=0x7ff75800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7567fc640 1 -- 172.21.15.16:0/553868602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75800cdb0 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.964+0000 7ff7567fc640 1 -- 172.21.15.16:0/553868602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff758005020 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.965+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7601a27f0 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.965+0000 7ff7567fc640 1 -- 172.21.15.16:0/553868602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7580116d0 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.965+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7601a8f50 con 0x7ff760109df0 2024-09-21T02:12:30.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.967+0000 7ff76db5f740 1 Processor -- start 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff7567fc640 1 -- 172.21.15.16:0/553868602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff758004b20 con 0x7ff760109df0 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 -- start start 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09c29510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c29a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a09b14dc0 con 0x557a09862840 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557a09ba11f0 con 0x557a0985a660 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff76db5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557a09b90620 con 0x557a09883520 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff764b86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff764b86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50910/0 (socket says 172.21.15.16:50910) 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff764b86640 1 -- 172.21.15.16:0/3427807487 learned_addr learned my addr 172.21.15.16:0/3427807487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff764b86640 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09c29510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.968+0000 7ff765387640 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c29a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 -- 172.21.15.16:0/3427807487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 msgr2=0x557a09c29510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09c29510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 -- 172.21.15.16:0/3427807487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 msgr2=0x557a09c29a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c29a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 -- 172.21.15.16:0/3427807487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a09b58fd0 con 0x557a09862840 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff764b86640 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff75000b6c0 tx=0x7ff75000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280d639f6dda16fd server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff7567fc640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 0x7ff74c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3427807487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750012d40 con 0x557a09862840 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff765387640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 0x7ff74c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3427807487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff750005020 con 0x557a09862840 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff74b7fe640 1 -- 172.21.15.16:0/3427807487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7500049c0 con 0x557a09862840 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.969+0000 7ff7567fc640 1 -- 172.21.15.16:0/553868602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff7580e7b90 con 0x7ff760109df0 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- 172.21.15.16:0/3427807487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 msgr2=0x557a09c29f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff75000b6c0 tx=0x7ff75000b650 comp rx=0 tx=0).stop 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff765387640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 0x7ff74c0b8c10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff75c000c00 tx=0x7ff75c005ed0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- 172.21.15.16:0/3427807487 shutdown_connections 2024-09-21T02:12:30.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c29f90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c29a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3427807487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09c29510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- 172.21.15.16:0/3427807487 >> 172.21.15.16:0/3427807487 conn(0x557a09877910 msgr2=0x557a09c27400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- 172.21.15.16:0/3427807487 shutdown_connections 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- 172.21.15.16:0/3427807487 wait complete. 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.970+0000 7ff76db5f740 1 -- start start 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 Processor -- start 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 0x557a09c34bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 0x557a09c40e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557a09b2b300 con 0x557a09862840 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557a09b94430 con 0x557a09883520 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557a099f1bb0 con 0x557a0985a660 2024-09-21T02:12:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff76db5f740 10 client.0 _ll_register_callbacks cb 0x557a09b45120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff764b86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff765387640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 0x557a09c34bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff764b86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50918/0 (socket says 172.21.15.16:50918) 2024-09-21T02:12:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff764b86640 1 -- 172.21.15.16:0/3467672603 learned_addr learned my addr 172.21.15.16:0/3467672603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.971+0000 7ff765b88640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 0x557a09c40e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 -- 172.21.15.16:0/3467672603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 msgr2=0x557a09c34bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 0x557a09c34bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 -- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 msgr2=0x557a09c40e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 0x557a09c40e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 -- 172.21.15.16:0/3467672603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a09b1a780 con 0x557a09862840 2024-09-21T02:12:30.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff765387640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 0x557a09c34bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff765b88640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 0x557a09c40e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff764b86640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff750012460 tx=0x7ff750004520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd4953e67d59399 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7500051d0 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff750005370 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff750005020 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.972+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff750005570 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a098eeb00 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff750005020 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7ff750026070 con 0x557a09862840 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 10 client.15945 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 20 client.15945 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:30.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09841ea0 0x557a09c46da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 4 client.15945 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a09ba3140 con 0x557a09841ea0 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.973+0000 7ff765387640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09841ea0 0x557a09c46da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.974+0000 7ff765387640 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09841ea0 0x557a09c46da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3484f1e739a5e8 server_cookie=93b8798e998971c2 in_seq=0 out_seq=0 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.974+0000 7ff748ff9640 10 client.15945 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff748ff9640 1 -- 172.21.15.16:0/3467672603 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557a09ba3140 con 0x557a09841ea0 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff748ff9640 10 client.15945 handle_command_reply: tid=0 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 client.15945 shutdown 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 msgr2=0x557a09c35100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff750012460 tx=0x7ff750004520 comp rx=0 tx=0).stop 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 shutdown_connections 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09841ea0 0x557a09c46da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09883520 0x557a09c40e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff748ff9640 0 client.15945 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:30.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09862840 0x557a09c35100 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 --2- 172.21.15.16:0/3467672603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a0985a660 0x557a09c34bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 >> 172.21.15.16:0/3467672603 conn(0x557a09877910 msgr2=0x557a09c2c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 shutdown_connections 2024-09-21T02:12:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 1 -- 172.21.15.16:0/3467672603 wait complete. 2024-09-21T02:12:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.975+0000 7ff76db5f740 20 client.15945 trim_cache size 0 max 16384 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.992+0000 7ff76db5f740 1 Processor -- start 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.992+0000 7ff76db5f740 1 -- start start 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c31db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09862840 0x557a09c322f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b2f80 con 0x557a09883520 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b4ac0 con 0x557a0985a660 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff76db5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b3650 con 0x557a09862840 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50926/0 (socket says 172.21.15.16:50926) 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 -- 172.21.15.16:0/788168144 learned_addr learned my addr 172.21.15.16:0/788168144 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff764b86640 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09862840 0x557a09c322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765387640 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c31db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 -- 172.21.15.16:0/788168144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09862840 msgr2=0x557a09c322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09862840 0x557a09c322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 -- 172.21.15.16:0/788168144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 msgr2=0x557a09c31db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c31db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 -- 172.21.15.16:0/788168144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a09965960 con 0x557a09883520 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff765b88640 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff7580092f0 tx=0x7ff7580acef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787ecc49b732af3c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.993+0000 7ff74affd640 1 -- 172.21.15.16:0/788168144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75801f070 con 0x557a09883520 2024-09-21T02:12:30.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff74affd640 1 -- 172.21.15.16:0/788168144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7580add70 con 0x557a09883520 2024-09-21T02:12:30.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff74affd640 1 -- 172.21.15.16:0/788168144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7580a8fc0 con 0x557a09883520 2024-09-21T02:12:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 -- 172.21.15.16:0/788168144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 msgr2=0x557a09c4c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff7580092f0 tx=0x7ff7580acef0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 -- 172.21.15.16:0/788168144 shutdown_connections 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a09883520 0x557a09c4c450 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09862840 0x557a09c322f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 --2- 172.21.15.16:0/788168144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a0985a660 0x557a09c31db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 -- 172.21.15.16:0/788168144 >> 172.21.15.16:0/788168144 conn(0x557a09877910 msgr2=0x557a09c360e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 -- 172.21.15.16:0/788168144 shutdown_connections 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.994+0000 7ff76db5f740 1 -- 172.21.15.16:0/788168144 wait complete. 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 -- start start 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 Processor -- start 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09862840 0x557a09c3bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09caa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b43f0 con 0x557a0985a660 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.995+0000 7ff76db5f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b3d20 con 0x557a09862840 2024-09-21T02:12:30.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff76db5f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7601b0f60 con 0x557a09883520 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff76db5f740 10 client.0 _ll_register_callbacks cb 0x7ff75c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50932/0 (socket says 172.21.15.16:50932) 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 -- 172.21.15.16:0/2365977430 learned_addr learned my addr 172.21.15.16:0/2365977430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff764b86640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09862840 0x557a09c3bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765b88640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09caa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 -- 172.21.15.16:0/2365977430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 msgr2=0x557a09caa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09caa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 -- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09862840 msgr2=0x557a09c3bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09862840 0x557a09c3bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 -- 172.21.15.16:0/2365977430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a09caad10 con 0x557a0985a660 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff765387640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff75c02a930 tx=0x7ff75c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5c4b4a79379ef9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c035040 con 0x557a0985a660 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff75c005240 con 0x557a0985a660 2024-09-21T02:12:31.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.996+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7580add70 con 0x557a0985a660 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff75c05f050 con 0x557a0985a660 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a09cb21a0 con 0x557a0985a660 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7580add70 con 0x557a0985a660 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7ff75c0035f0 con 0x557a0985a660 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 10 client.15951 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 20 client.15951 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09817250 0x557a09cb4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 4 client.15951 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ec7be942", "release": true, "await": true}] 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ec7be942", "release": true, "await": true}) -- 0x557a09b9ebf0 con 0x557a09817250 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.997+0000 7ff764b86640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09817250 0x557a09cb4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.998+0000 7ff764b86640 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09817250 0x557a09cb4990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6757797fa2770f9a server_cookie=c00979225bbcbf4c in_seq=0 out_seq=0 2024-09-21T02:12:31.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:30.998+0000 7ff748ff9640 10 client.15951 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff748ff9640 1 -- 172.21.15.16:0/2365977430 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557a09b9ebf0 con 0x557a09817250 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff748ff9640 10 client.15951 handle_command_reply: tid=0 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 client.15951 shutdown 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 msgr2=0x557a09c3b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff75c02a930 tx=0x7ff75c009260 comp rx=0 tx=0).stop 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 shutdown_connections 2024-09-21T02:12:31.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x557a09817250 0x557a09cb4990 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557a09883520 0x557a09caa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557a09862840 0x557a09c3bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 --2- 172.21.15.16:0/2365977430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557a0985a660 0x557a09c3b7f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 >> 172.21.15.16:0/2365977430 conn(0x557a09877910 msgr2=0x557a09c33440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff748ff9640 0 client.15951 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 shutdown_connections 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 1 -- 172.21.15.16:0/2365977430 wait complete. 2024-09-21T02:12:31.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.003+0000 7ff76db5f740 20 client.15951 trim_cache size 0 max 16384 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 20, 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:12:31.007 INFO:teuthology.orchestra.run.smithi016.stdout: "ec7be942": { 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 20, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 130.0, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:12:31.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.005+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 msgr2=0x7ff74c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.005+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 0x7ff74c0b8c10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff75c000c00 tx=0x7ff75c005ed0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.005+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 msgr2=0x7ff7601a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.005+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff75800eaa0 tx=0x7ff75800ef60 comp rx=0 tx=0).stop 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 shutdown_connections 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff74c0b6700 0x7ff74c0b8c10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff760109df0 0x7ff7601a1dd0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7601094a0 0x7ff76019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 --2- 172.21.15.16:0/553868602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff760103b60 0x7ff76019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.006+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 >> 172.21.15.16:0/553868602 conn(0x7ff76007b850 msgr2=0x7ff760107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.007+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 shutdown_connections 2024-09-21T02:12:31.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:31.007+0000 7ff7675eb640 1 -- 172.21.15.16:0/553868602 wait complete. 2024-09-21T02:12:31.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:31 smithi123 ceph-mon[33453]: pgmap v513: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 574 op/s 2024-09-21T02:12:31.640 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:31 smithi016 ceph-mon[31245]: pgmap v513: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 574 op/s 2024-09-21T02:12:31.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:31 smithi161 ceph-mon[34061]: pgmap v513: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 35 MiB/s wr, 574 op/s 2024-09-21T02:12:31.860 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ec7be942', total seconds elapsed: 20.8 2024-09-21T02:12:33.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:33 smithi123 ceph-mon[33453]: pgmap v514: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 30 MiB/s wr, 486 op/s 2024-09-21T02:12:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:33 smithi016 ceph-mon[31245]: pgmap v514: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 30 MiB/s wr, 486 op/s 2024-09-21T02:12:33.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:33 smithi161 ceph-mon[34061]: pgmap v514: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 30 MiB/s wr, 486 op/s 2024-09-21T02:12:35.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:35 smithi123 ceph-mon[33453]: pgmap v515: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 30 MiB/s wr, 491 op/s 2024-09-21T02:12:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:35 smithi016 ceph-mon[31245]: pgmap v515: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 30 MiB/s wr, 491 op/s 2024-09-21T02:12:35.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:35 smithi161 ceph-mon[34061]: pgmap v515: 129 pgs: 129 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 30 MiB/s wr, 491 op/s 2024-09-21T02:12:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:37 smithi123 ceph-mon[33453]: pgmap v516: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 420 op/s 2024-09-21T02:12:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:37 smithi016 ceph-mon[31245]: pgmap v516: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 420 op/s 2024-09-21T02:12:37.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:37 smithi161 ceph-mon[34061]: pgmap v516: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 420 op/s 2024-09-21T02:12:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:39 smithi123 ceph-mon[33453]: pgmap v517: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 431 op/s 2024-09-21T02:12:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:39 smithi016 ceph-mon[31245]: pgmap v517: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 431 op/s 2024-09-21T02:12:39.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:39 smithi161 ceph-mon[34061]: pgmap v517: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 431 op/s 2024-09-21T02:12:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:41 smithi123 ceph-mon[33453]: pgmap v518: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 509 op/s 2024-09-21T02:12:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:41 smithi016 ceph-mon[31245]: pgmap v518: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 509 op/s 2024-09-21T02:12:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:41 smithi161 ceph-mon[34061]: pgmap v518: 129 pgs: 129 active+clean; 14 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 509 op/s 2024-09-21T02:12:42.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:42 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:12:42.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:42 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:12:42.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:42 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:12:43.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:43 smithi016 ceph-mon[31245]: pgmap v519: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 34 MiB/s wr, 558 op/s 2024-09-21T02:12:43.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:12:43.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:12:43.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:12:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:43 smithi161 ceph-mon[34061]: pgmap v519: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 34 MiB/s wr, 558 op/s 2024-09-21T02:12:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:12:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:12:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:12:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:43 smithi123 ceph-mon[33453]: pgmap v519: 129 pgs: 129 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 34 MiB/s wr, 558 op/s 2024-09-21T02:12:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:12:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:12:43.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:12:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:45 smithi016 ceph-mon[31245]: pgmap v520: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T02:12:45.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:45 smithi161 ceph-mon[34061]: pgmap v520: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T02:12:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:45 smithi123 ceph-mon[33453]: pgmap v520: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T02:12:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:47 smithi016 ceph-mon[31245]: pgmap v521: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 698 op/s 2024-09-21T02:12:47.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:47 smithi161 ceph-mon[34061]: pgmap v521: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 698 op/s 2024-09-21T02:12:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:47 smithi123 ceph-mon[33453]: pgmap v521: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 698 op/s 2024-09-21T02:12:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:49 smithi016 ceph-mon[31245]: pgmap v522: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 699 op/s 2024-09-21T02:12:49.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:49 smithi161 ceph-mon[34061]: pgmap v522: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 699 op/s 2024-09-21T02:12:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:49 smithi123 ceph-mon[33453]: pgmap v522: 129 pgs: 129 active+clean; 15 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 43 MiB/s wr, 699 op/s 2024-09-21T02:12:50.433 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:12:50.433 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T02:12:50.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:12:51.037 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:51.519 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:51 smithi016 ceph-mon[31245]: pgmap v523: 129 pgs: 129 active+clean; 15 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 47 MiB/s wr, 754 op/s 2024-09-21T02:12:51.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:51 smithi161 ceph-mon[34061]: pgmap v523: 129 pgs: 129 active+clean; 15 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 47 MiB/s wr, 754 op/s 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 -- 172.21.15.16:0/523087558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 msgr2=0x7f3f4c06d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 --2- 172.21.15.16:0/523087558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c06d5e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009900 tx=0x7f3f3c02a420 comp rx=0 tx=0).stop 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 -- 172.21.15.16:0/523087558 shutdown_connections 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 --2- 172.21.15.16:0/523087558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 0x7f3f4c073c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 --2- 172.21.15.16:0/523087558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f4c072b10 0x7f3f4c072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 --2- 172.21.15.16:0/523087558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c06d5e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 -- 172.21.15.16:0/523087558 >> 172.21.15.16:0/523087558 conn(0x7f3f4c0fe2c0 msgr2=0x7f3f4c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.688+0000 7f3f51cfd640 1 -- 172.21.15.16:0/523087558 shutdown_connections 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.689+0000 7f3f51cfd640 1 -- 172.21.15.16:0/523087558 wait complete. 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 Processor -- start 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 -- start start 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f4c072b10 0x7f3f4c1a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 0x7f3f4c1ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c11af60 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c11ade0 con 0x7f3f4c073460 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.690+0000 7f3f51cfd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c11b0e0 con 0x7f3f4c072b10 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54418/0 (socket says 172.21.15.16:54418) 2024-09-21T02:12:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 -- 172.21.15.16:0/4291435187 learned_addr learned my addr 172.21.15.16:0/4291435187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f514fc640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 0x7f3f4c1ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 -- 172.21.15.16:0/4291435187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f4c072b10 msgr2=0x7f3f4c1a9700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f4c072b10 0x7f3f4c1a9700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 -- 172.21.15.16:0/4291435187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 msgr2=0x7f3f4c1ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 0x7f3f4c1ae9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.691+0000 7f3f50cfb640 1 -- 172.21.15.16:0/4291435187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c1af0d0 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.692+0000 7f3f50cfb640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005ab0 tx=0x7f3f3c003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.694+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c03f070 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.694+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3f3c03bd00 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.694+0000 7f3f51cfd640 1 -- 172.21.15.16:0/4291435187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c1af360 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.695+0000 7f3f51cfd640 1 -- 172.21.15.16:0/4291435187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c1b5b20 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.695+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c035030 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.699+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4c072fa0 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.716+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3f3c042070 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.718+0000 7f3f49ffb640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 0x7f3f340b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.718+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f3f3c108ce0 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.719+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3f3c109180 con 0x7f3f4c06d1d0 2024-09-21T02:12:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.725+0000 7f3f4bfff640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 0x7f3f340b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:51.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:51.729+0000 7f3f4bfff640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 0x7f3f340b8d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c073bd0 tx=0x7f3f40006bd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:51 smithi123 ceph-mon[33453]: pgmap v523: 129 pgs: 129 active+clean; 15 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 47 MiB/s wr, 754 op/s 2024-09-21T02:12:52.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.187+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f4c063fc0 con 0x7f3f4c06d1d0 2024-09-21T02:12:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.189+0000 7f3f49ffb640 1 -- 172.21.15.16:0/4291435187 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f3f3c0ceee0 con 0x7f3f4c06d1d0 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.193+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 msgr2=0x7f3f340b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.193+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 0x7f3f340b8d90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c073bd0 tx=0x7f3f40006bd0 comp rx=0 tx=0).stop 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.193+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 msgr2=0x7f3f4c1a9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:52.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.193+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005ab0 tx=0x7f3f3c003d80 comp rx=0 tx=0).stop 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 shutdown_connections 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3f340b6880 0x7f3f340b8d90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f4c073460 0x7f3f4c1ae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f4c072b10 0x7f3f4c1a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 --2- 172.21.15.16:0/4291435187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4c06d1d0 0x7f3f4c1a9160 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.196+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 >> 172.21.15.16:0/4291435187 conn(0x7f3f4c0fe2c0 msgr2=0x7f3f4c06ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.201+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 shutdown_connections 2024-09-21T02:12:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:52.203+0000 7f3f337fe640 1 -- 172.21.15.16:0/4291435187 wait complete. 2024-09-21T02:12:52.575 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:52 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4291435187' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:52.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:52 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4291435187' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:52.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:52 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4291435187' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:12:53.069 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T02:12:53.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T02:12:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:12:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:53 smithi016 ceph-mon[31245]: pgmap v524: 129 pgs: 129 active+clean; 15 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T02:12:53.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:53 smithi161 ceph-mon[34061]: pgmap v524: 129 pgs: 129 active+clean; 15 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T02:12:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:53 smithi123 ceph-mon[33453]: pgmap v524: 129 pgs: 129 active+clean; 15 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.303+0000 7f44f3fff640 1 -- 172.21.15.16:0/91176497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41081f0 msgr2=0x7f44f4108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.303+0000 7f44f3fff640 1 --2- 172.21.15.16:0/91176497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41081f0 0x7f44f4108600 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009980 tx=0x7f44e402e850 comp rx=0 tx=0).stop 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 -- 172.21.15.16:0/91176497 shutdown_connections 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 --2- 172.21.15.16:0/91176497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f410f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 --2- 172.21.15.16:0/91176497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41081f0 0x7f44f4108600 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 --2- 172.21.15.16:0/91176497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41028b0 0x7f44f4102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 -- 172.21.15.16:0/91176497 >> 172.21.15.16:0/91176497 conn(0x7f44f40fe2e0 msgr2=0x7f44f4100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 -- 172.21.15.16:0/91176497 shutdown_connections 2024-09-21T02:12:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.304+0000 7f44f3fff640 1 -- 172.21.15.16:0/91176497 wait complete. 2024-09-21T02:12:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.305+0000 7f44f3fff640 1 Processor -- start 2024-09-21T02:12:54.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.305+0000 7f44f3fff640 1 -- start start 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41028b0 0x7f44f419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44f4111630 con 0x7f44f41028b0 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44f41114b0 con 0x7f44f41081f0 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.306+0000 7f44f3fff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44f41117b0 con 0x7f44f4108b40 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.308+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.308+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:38686/0 (socket says 172.21.15.16:38686) 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.308+0000 7f44ebfff640 1 -- 172.21.15.16:0/1795825180 learned_addr learned my addr 172.21.15.16:0/1795825180 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.308+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41028b0 0x7f44f419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.308+0000 7f44f37fe640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 -- 172.21.15.16:0/1795825180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 msgr2=0x7f44f4197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 -- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41028b0 msgr2=0x7f44f419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41028b0 0x7f44f419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 -- 172.21.15.16:0/1795825180 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f4198c70 con 0x7f44f4108b40 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44ebfff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:12:54.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f37fe640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f4198450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00be00 tx=0x7f44ec00bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.310+0000 7f44f0ff9640 1 -- 172.21.15.16:0/1795825180 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec019070 con 0x7f44f4108b40 2024-09-21T02:12:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.311+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f419d8f0 con 0x7f44f4108b40 2024-09-21T02:12:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.311+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f419de80 con 0x7f44f4108b40 2024-09-21T02:12:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.311+0000 7f44f0ff9640 1 -- 172.21.15.16:0/1795825180 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44ec0148d0 con 0x7f44f4108b40 2024-09-21T02:12:54.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.311+0000 7f44f0ff9640 1 -- 172.21.15.16:0/1795825180 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec012840 con 0x7f44f4108b40 2024-09-21T02:12:54.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.314+0000 7f44ff9af740 1 Processor -- start 2024-09-21T02:12:54.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.317+0000 7f44f0ff9640 1 -- 172.21.15.16:0/1795825180 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f44ec01c070 con 0x7f44f4108b40 2024-09-21T02:12:54.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.317+0000 7f44ff9af740 1 -- start start 2024-09-21T02:12:54.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3a985c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3a98b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3a0ebb0 con 0x55f2a36c8990 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f2a39fdfe0 con 0x55f2a36f1850 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ff9af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3a01df0 con 0x55f2a36d0b70 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:38694/0 (socket says 172.21.15.16:38694) 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ebfff640 1 -- 172.21.15.16:0/527876978 learned_addr learned my addr 172.21.15.16:0/527876978 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.318+0000 7f44ebfff640 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3a98b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.319+0000 7f44ebfff640 1 -- 172.21.15.16:0/527876978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 msgr2=0x55f2a3a98b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:54.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.319+0000 7f44ebfff640 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3a98b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.319+0000 7f44ebfff640 1 -- 172.21.15.16:0/527876978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 msgr2=0x55f2a3a985c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:12:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.319+0000 7f44ebfff640 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3a985c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.319+0000 7f44ebfff640 1 -- 172.21.15.16:0/527876978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a376bba0 con 0x55f2a36f1850 2024-09-21T02:12:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.320+0000 7f44f0ff9640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 0x7f44e00b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.320+0000 7f44f0ff9640 1 -- 172.21.15.16:0/1795825180 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f44ec0e6ee0 con 0x7f44f4108b40 2024-09-21T02:12:54.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.324+0000 7f44ebfff640 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44e40096c0 tx=0x7f44e402e850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea89ab483e7665d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.325+0000 7f44e97fa640 1 -- 172.21.15.16:0/527876978 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e403a040 con 0x55f2a36f1850 2024-09-21T02:12:54.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.325+0000 7f44e97fa640 1 -- 172.21.15.16:0/527876978 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44e4003c50 con 0x55f2a36f1850 2024-09-21T02:12:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.326+0000 7f44ff9af740 1 -- 172.21.15.16:0/527876978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 msgr2=0x55f2a3a98080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.326+0000 7f44ff9af740 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44e40096c0 tx=0x7f44e402e850 comp rx=0 tx=0).stop 2024-09-21T02:12:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.328+0000 7f44ff9af740 1 -- 172.21.15.16:0/527876978 shutdown_connections 2024-09-21T02:12:54.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.328+0000 7f44ff9af740 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3a98b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.328+0000 7f44ff9af740 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3a985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.328+0000 7f44ff9af740 1 --2- 172.21.15.16:0/527876978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3a98080 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.328+0000 7f44ff9af740 1 -- 172.21.15.16:0/527876978 >> 172.21.15.16:0/527876978 conn(0x55f2a36e5c40 msgr2=0x55f2a3a95f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.331+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 0x7f44e00b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.335+0000 7f44ff9af740 1 -- 172.21.15.16:0/527876978 shutdown_connections 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.337+0000 7f44ff9af740 1 -- 172.21.15.16:0/527876978 wait complete. 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 -- start start 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 Processor -- start 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3ac0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2a385f7b0 con 0x55f2a36c8990 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3998e90 con 0x55f2a36f1850 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3aa68a0 con 0x55f2a36d0b70 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.338+0000 7f44ff9af740 10 client.0 _ll_register_callbacks cb 0x55f2a39fb710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.339+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 0x7f44e00b8ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b3c0 tx=0x7f44dc00d040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.340+0000 7f44f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.340+0000 7f44f2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54456/0 (socket says 172.21.15.16:54456) 2024-09-21T02:12:54.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.340+0000 7f44f2ffd640 1 -- 172.21.15.16:0/3140815006 learned_addr learned my addr 172.21.15.16:0/3140815006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.343+0000 7f44f2ffd640 1 -- 172.21.15.16:0/3140815006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 msgr2=0x55f2a3aa40f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:12:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.343+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa40f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.343+0000 7f44f2ffd640 1 -- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 msgr2=0x55f2a3ac0c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:12:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.343+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3ac0c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.343+0000 7f44f2ffd640 1 -- 172.21.15.16:0/3140815006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a3ac11c0 con 0x55f2a36c8990 2024-09-21T02:12:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.347+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f44dc018730 tx=0x7f44dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12d5e3c0e5947fc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.348+0000 7f44da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:12:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.349+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc005020 con 0x55f2a36c8990 2024-09-21T02:12:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.349+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f44dc0051c0 con 0x55f2a36c8990 2024-09-21T02:12:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.349+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e4003c50 con 0x55f2a36c8990 2024-09-21T02:12:54.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.352+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44dc0053c0 con 0x55f2a36c8990 2024-09-21T02:12:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.357+0000 7f44ff9af740 10 client.15966 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:54.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.357+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2a3aa8510 con 0x55f2a36c8990 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44e4003c50 con 0x55f2a36c8990 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f44dc0169a0 con 0x55f2a36c8990 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 10 client.15966 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 10 client.15966 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 20 client.15966 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a36b01d0 0x55f2a3aabfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 4 client.15966 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:12:54.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.360+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a3a10b00 con 0x55f2a36b01d0 2024-09-21T02:12:54.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.363+0000 7f44ebfff640 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a36b01d0 0x55f2a3aabfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.369+0000 7f44ebfff640 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a36b01d0 0x55f2a3aabfa0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7533c4815eb0e0c server_cookie=3d31061ac64c56df in_seq=0 out_seq=0 2024-09-21T02:12:54.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.369+0000 7f44da7fc640 10 client.15966 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:54.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.375+0000 7f44da7fc640 1 -- 172.21.15.16:0/3140815006 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f2a3a10b00 con 0x55f2a36b01d0 2024-09-21T02:12:54.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.375+0000 7f44da7fc640 10 client.15966 handle_command_reply: tid=0 2024-09-21T02:12:54.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.378+0000 7f44ff9af740 1 client.15966 shutdown 2024-09-21T02:12:54.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.378+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 msgr2=0x55f2a3aa3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.378+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f44dc018730 tx=0x7f44dc0046c0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 shutdown_connections 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a36b01d0 0x55f2a3aabfa0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36f1850 0x55f2a3ac0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 --2- 172.21.15.16:0/3140815006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36c8990 0x55f2a3aa3bb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 >> 172.21.15.16:0/3140815006 conn(0x55f2a36e5c40 msgr2=0x55f2a3a9b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.379+0000 7f44da7fc640 0 client.15966 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.382+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 shutdown_connections 2024-09-21T02:12:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.382+0000 7f44ff9af740 1 -- 172.21.15.16:0/3140815006 wait complete. 2024-09-21T02:12:54.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.382+0000 7f44ff9af740 20 client.15966 trim_cache size 0 max 16384 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.418+0000 7f44ff9af740 1 Processor -- start 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.418+0000 7f44ff9af740 1 -- start start 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36e5c40 0x55f2a3a9e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:38702/0 (socket says 172.21.15.16:38702) 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.419+0000 7f44ebfff640 1 -- 172.21.15.16:0/1114077244 learned_addr learned my addr 172.21.15.16:0/1114077244 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4010030 con 0x55f2a36e5c40 2024-09-21T02:12:54.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4011b70 con 0x55f2a36c8990 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4010700 con 0x55f2a36d0b70 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ebfff640 1 -- 172.21.15.16:0/1114077244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 msgr2=0x55f2a3aa0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ebfff640 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ebfff640 1 -- 172.21.15.16:0/1114077244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36e5c40 msgr2=0x55f2a3a9e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ebfff640 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36e5c40 0x55f2a3a9e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.420+0000 7f44ebfff640 1 -- 172.21.15.16:0/1114077244 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a3a9f240 con 0x55f2a36c8990 2024-09-21T02:12:54.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.421+0000 7f44ebfff640 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44e402ed70 tx=0x7f44e4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46247522ee7c13aa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.424+0000 7f44e8ff9640 1 -- 172.21.15.16:0/1114077244 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e403a040 con 0x55f2a36c8990 2024-09-21T02:12:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.424+0000 7f44e8ff9640 1 -- 172.21.15.16:0/1114077244 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44e4032050 con 0x55f2a36c8990 2024-09-21T02:12:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.424+0000 7f44e8ff9640 1 -- 172.21.15.16:0/1114077244 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e40366a0 con 0x55f2a36c8990 2024-09-21T02:12:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 msgr2=0x55f2a3a95840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44e402ed70 tx=0x7f44e4005850 comp rx=0 tx=0).stop 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 shutdown_connections 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3aa0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36c8990 0x55f2a3a95840 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44e402ed70 tx=0x7f44e4005850 comp rx=0 tx=0).stop 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 --2- 172.21.15.16:0/1114077244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36e5c40 0x55f2a3a9e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.426+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 >> 172.21.15.16:0/1114077244 conn(0x55f2a3a95f20 msgr2=0x55f2a3aa2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.428+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 shutdown_connections 2024-09-21T02:12:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.429+0000 7f44ff9af740 1 -- 172.21.15.16:0/1114077244 wait complete. 2024-09-21T02:12:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.430+0000 7f44ff9af740 1 -- start start 2024-09-21T02:12:54.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.430+0000 7f44ff9af740 1 Processor -- start 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.430+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36e5c40 0x55f2a3ab0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36f1850 0x55f2a3b42460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d40114a0 con 0x55f2a36f1850 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4010dd0 con 0x55f2a36e5c40 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44d400e010 con 0x55f2a36d0b70 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44ff9af740 10 client.0 _ll_register_callbacks cb 0x7f44dc00bd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44f2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44f2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:42848/0 (socket says 172.21.15.16:42848) 2024-09-21T02:12:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.431+0000 7f44f2ffd640 1 -- 172.21.15.16:0/758147102 learned_addr learned my addr 172.21.15.16:0/758147102 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f37fe640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36f1850 0x55f2a3b42460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44ebfff640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36e5c40 0x55f2a3ab0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f2ffd640 1 -- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36e5c40 msgr2=0x55f2a3ab0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36e5c40 0x55f2a3ab0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f2ffd640 1 -- 172.21.15.16:0/758147102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36f1850 msgr2=0x55f2a3b42460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36f1850 0x55f2a3b42460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.432+0000 7f44f2ffd640 1 -- 172.21.15.16:0/758147102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a3ab1790 con 0x55f2a36d0b70 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.433+0000 7f44f2ffd640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f44dc016fd0 tx=0x7f44dc016730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9cf30eb5c06558 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:12:54.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.433+0000 7f44da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:12:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.434+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc036020 con 0x55f2a36d0b70 2024-09-21T02:12:54.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.434+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f44dc0237b0 con 0x55f2a36d0b70 2024-09-21T02:12:54.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.438+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e4032050 con 0x55f2a36d0b70 2024-09-21T02:12:54.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.442+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44dc0239b0 con 0x55f2a36d0b70 2024-09-21T02:12:54.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.442+0000 7f44ff9af740 10 client.25645 fetch_fsmap learned FSMap version 28 2024-09-21T02:12:54.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.442+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2a3b43930 con 0x55f2a36d0b70 2024-09-21T02:12:54.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.447+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44e4032050 con 0x55f2a36d0b70 2024-09-21T02:12:54.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f44dc032b90 con 0x55f2a36d0b70 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 10 client.25645 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 10 client.25645 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 20 client.25645 populate_metadata read hostname 'smithi016' 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a3685580 0x55f2a3b46140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 4 client.25645 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.448+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x55f2a3a0c5b0 con 0x55f2a3685580 2024-09-21T02:12:54.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.449+0000 7f44ebfff640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a3685580 0x55f2a3b46140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:12:54.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.450+0000 7f44ebfff640 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a3685580 0x55f2a3b46140 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460f270e015ec282 server_cookie=9ae99104ccbbe4e4 in_seq=0 out_seq=0 2024-09-21T02:12:54.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.451+0000 7f44da7fc640 10 client.25645 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44da7fc640 1 -- 172.21.15.16:0/758147102 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f2a3a0c5b0 con 0x55f2a3685580 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44da7fc640 10 client.25645 handle_command_reply: tid=0 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 client.25645 shutdown 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 msgr2=0x55f2a3ab0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f44dc016fd0 tx=0x7f44dc016730 comp rx=0 tx=0).stop 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 shutdown_connections 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2a3685580 0x55f2a3b46140 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2a36f1850 0x55f2a3b42460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2a36e5c40 0x55f2a3ab0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 --2- 172.21.15.16:0/758147102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2a36d0b70 0x55f2a3ab0a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 >> 172.21.15.16:0/758147102 conn(0x55f2a3a95f20 msgr2=0x55f2a3aa2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.516+0000 7f44da7fc640 0 client.25645 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.517+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 shutdown_connections 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.517+0000 7f44ff9af740 1 -- 172.21.15.16:0/758147102 wait complete. 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.517+0000 7f44ff9af740 20 client.25645 trim_cache size 0 max 16384 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:12:54.519 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 22, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "c96ba1ec": { 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 22, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.2, 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:12:54.520 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:12:54.521 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 msgr2=0x7f44e00b8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 0x7f44e00b8ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b3c0 tx=0x7f44dc00d040 comp rx=0 tx=0).stop 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 msgr2=0x7f44f4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f4198450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00be00 tx=0x7f44ec00bef0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 shutdown_connections 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f44e00b67d0 0x7f44e00b8ce0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44f4108b40 0x7f44f4198450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44f41081f0 0x7f44f4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 --2- 172.21.15.16:0/1795825180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f41028b0 0x7f44f419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:12:54.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 >> 172.21.15.16:0/1795825180 conn(0x7f44f40fe2e0 msgr2=0x7f44f4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:12:54.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 shutdown_connections 2024-09-21T02:12:54.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:12:54.518+0000 7f44f3fff640 1 -- 172.21.15.16:0/1795825180 wait complete. 2024-09-21T02:12:55.358 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c96ba1ec', quiesce took 4.9 seconds. Will release after: 14.35 2024-09-21T02:12:55.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:55 smithi016 ceph-mon[31245]: pgmap v525: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-21T02:12:55.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:55 smithi161 ceph-mon[34061]: pgmap v525: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-21T02:12:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:55 smithi123 ceph-mon[33453]: pgmap v525: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 48 MiB/s wr, 778 op/s 2024-09-21T02:12:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:57 smithi016 ceph-mon[31245]: pgmap v526: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 40 MiB/s wr, 647 op/s 2024-09-21T02:12:57.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:57 smithi161 ceph-mon[34061]: pgmap v526: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 40 MiB/s wr, 647 op/s 2024-09-21T02:12:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:57 smithi123 ceph-mon[33453]: pgmap v526: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 40 MiB/s wr, 647 op/s 2024-09-21T02:12:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:12:59 smithi016 ceph-mon[31245]: pgmap v527: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-21T02:12:59.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:12:59 smithi161 ceph-mon[34061]: pgmap v527: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-21T02:12:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:12:59 smithi123 ceph-mon[33453]: pgmap v527: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 38 MiB/s wr, 609 op/s 2024-09-21T02:13:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:01 smithi016 ceph-mon[31245]: pgmap v528: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 34 MiB/s wr, 551 op/s 2024-09-21T02:13:01.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:01 smithi161 ceph-mon[34061]: pgmap v528: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 34 MiB/s wr, 551 op/s 2024-09-21T02:13:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:01 smithi123 ceph-mon[33453]: pgmap v528: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 34 MiB/s wr, 551 op/s 2024-09-21T02:13:03.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:03 smithi016 ceph-mon[31245]: pgmap v529: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 416 op/s 2024-09-21T02:13:03.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:03 smithi161 ceph-mon[34061]: pgmap v529: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 416 op/s 2024-09-21T02:13:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:03 smithi123 ceph-mon[33453]: pgmap v529: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 MiB/s wr, 416 op/s 2024-09-21T02:13:05.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:05 smithi016 ceph-mon[31245]: pgmap v530: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 MiB/s wr, 318 op/s 2024-09-21T02:13:05.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:05 smithi161 ceph-mon[34061]: pgmap v530: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 MiB/s wr, 318 op/s 2024-09-21T02:13:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:05 smithi123 ceph-mon[33453]: pgmap v530: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 18 MiB/s wr, 318 op/s 2024-09-21T02:13:07.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:07 smithi016 ceph-mon[31245]: pgmap v531: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.4 MiB/s wr, 142 op/s 2024-09-21T02:13:07.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:07 smithi161 ceph-mon[34061]: pgmap v531: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.4 MiB/s wr, 142 op/s 2024-09-21T02:13:07.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:07 smithi123 ceph-mon[33453]: pgmap v531: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.4 MiB/s wr, 142 op/s 2024-09-21T02:13:09.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:09 smithi016 ceph-mon[31245]: pgmap v532: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 90 op/s 2024-09-21T02:13:09.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:09 smithi161 ceph-mon[34061]: pgmap v532: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 90 op/s 2024-09-21T02:13:09.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:13:09.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:09 smithi123 ceph-mon[33453]: pgmap v532: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 90 op/s 2024-09-21T02:13:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 -- 172.21.15.16:0/3432706265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 msgr2=0x7f9cec101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 --2- 172.21.15.16:0/3432706265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec101fd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009900 tx=0x7f9cdc02a480 comp rx=0 tx=0).stop 2024-09-21T02:13:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 -- 172.21.15.16:0/3432706265 shutdown_connections 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 --2- 172.21.15.16:0/3432706265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 0x7f9cec110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 --2- 172.21.15.16:0/3432706265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 0x7f9cec109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 --2- 172.21.15.16:0/3432706265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec101fd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 -- 172.21.15.16:0/3432706265 >> 172.21.15.16:0/3432706265 conn(0x7f9cec077800 msgr2=0x7f9cec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 -- 172.21.15.16:0/3432706265 shutdown_connections 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.502+0000 7f9cf0853640 1 -- 172.21.15.16:0/3432706265 wait complete. 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 Processor -- start 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 -- start start 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 0x7f9cec103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 0x7f9cec103e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113930 con 0x7f9cec101bc0 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1137b0 con 0x7f9cec109d50 2024-09-21T02:13:10.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cf0853640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113ab0 con 0x7f9cec102510 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cea59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36772/0 (socket says 172.21.15.16:36772) 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cea59c640 1 -- 172.21.15.16:0/2885816072 learned_addr learned my addr 172.21.15.16:0/2885816072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9cead9d640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 0x7f9cec103e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.503+0000 7f9ce9d9b640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 0x7f9cec103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 -- 172.21.15.16:0/2885816072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 msgr2=0x7f9cec103930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 0x7f9cec103930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 -- 172.21.15.16:0/2885816072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 msgr2=0x7f9cec103e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 0x7f9cec103e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 -- 172.21.15.16:0/2885816072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec1a8640 con 0x7f9cec101bc0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cea59c640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009230 tx=0x7f9cdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc03f030 con 0x7f9cec101bc0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9cdc003a80 con 0x7f9cec101bc0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec1a88d0 con 0x7f9cec101bc0 2024-09-21T02:13:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc035070 con 0x7f9cec101bc0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.504+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1a8db0 con 0x7f9cec101bc0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.505+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec101fd0 con 0x7f9cec101bc0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.507+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9cdc042030 con 0x7f9cec101bc0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.508+0000 7f9cdb7fe640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 0x7f9cd40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.508+0000 7f9ce9d9b640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 0x7f9cd40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.508+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f9cdc107ff0 con 0x7f9cec101bc0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.508+0000 7f9ce9d9b640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 0x7f9cd40b8b90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0005fd0 tx=0x7f9ce0005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.513+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9cdc0ce1d0 con 0x7f9cec101bc0 2024-09-21T02:13:10.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.736+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cec02e640 con 0x7f9cec101bc0 2024-09-21T02:13:10.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.737+0000 7f9cdb7fe640 1 -- 172.21.15.16:0/2885816072 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f9cdc0d25b0 con 0x7f9cec101bc0 2024-09-21T02:13:10.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:13:10.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:13:10.738 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 msgr2=0x7f9cd40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 0x7f9cd40b8b90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0005fd0 tx=0x7f9ce0005850 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 msgr2=0x7f9cec1033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009230 tx=0x7f9cdc004270 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 shutdown_connections 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9cd40b6680 0x7f9cd40b8b90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cec109d50 0x7f9cec103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cec102510 0x7f9cec103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 --2- 172.21.15.16:0/2885816072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cec101bc0 0x7f9cec1033f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 >> 172.21.15.16:0/2885816072 conn(0x7f9cec077800 msgr2=0x7f9cec100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:10.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 shutdown_connections 2024-09-21T02:13:10.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:10.739+0000 7f9cf0853640 1 -- 172.21.15.16:0/2885816072 wait complete. 2024-09-21T02:13:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:11 smithi016 ceph-mon[31245]: pgmap v533: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-09-21T02:13:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:11 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2885816072' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:11.534 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id c96ba1ec --release --await' 2024-09-21T02:13:11.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id c96ba1ec --release --await 2024-09-21T02:13:11.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:11 smithi161 ceph-mon[34061]: pgmap v533: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-09-21T02:13:11.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:11 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2885816072' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:11 smithi123 ceph-mon[33453]: pgmap v533: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.8 MiB/s wr, 55 op/s 2024-09-21T02:13:11.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:11 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2885816072' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- 172.21.15.16:0/1639847084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 msgr2=0x7f9620110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 --2- 172.21.15.16:0/1639847084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f9620110320 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9618009ca0 tx=0x7f961802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- 172.21.15.16:0/1639847084 shutdown_connections 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 --2- 172.21.15.16:0/1639847084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f9620110320 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 --2- 172.21.15.16:0/1639847084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9620105320 0x7f962010d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 --2- 172.21.15.16:0/1639847084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96201049d0 0x7f9620104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- 172.21.15.16:0/1639847084 >> 172.21.15.16:0/1639847084 conn(0x7f9620079740 msgr2=0x7f9620079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- 172.21.15.16:0/1639847084 shutdown_connections 2024-09-21T02:13:12.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- 172.21.15.16:0/1639847084 wait complete. 2024-09-21T02:13:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 Processor -- start 2024-09-21T02:13:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.305+0000 7f9627213640 1 -- start start 2024-09-21T02:13:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96201049d0 0x7f962019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9620105320 0x7f962019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9620113990 con 0x7f962010de30 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9620113810 con 0x7f9620105320 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9627213640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9620113b10 con 0x7f96201049d0 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36784/0 (socket says 172.21.15.16:36784) 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 -- 172.21.15.16:0/2076978466 learned_addr learned my addr 172.21.15.16:0/2076978466 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9617fff640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9620105320 0x7f962019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f9624faf640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96201049d0 0x7f962019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 -- 172.21.15.16:0/2076978466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96201049d0 msgr2=0x7f962019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96201049d0 0x7f962019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 -- 172.21.15.16:0/2076978466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9620105320 msgr2=0x7f962019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9620105320 0x7f962019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.306+0000 7f96257b0640 1 -- 172.21.15.16:0/2076978466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96201a02d0 con 0x7f962010de30 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f96257b0640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9618005cc0 tx=0x7f9618032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f9615ffb640 1 -- 172.21.15.16:0/2076978466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618032e00 con 0x7f962010de30 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f9615ffb640 1 -- 172.21.15.16:0/2076978466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96180044d0 con 0x7f962010de30 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96201a0560 con 0x7f962010de30 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f9615ffb640 1 -- 172.21.15.16:0/2076978466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618030b70 con 0x7f962010de30 2024-09-21T02:13:12.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.307+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96201a6d20 con 0x7f962010de30 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f9615ffb640 1 -- 172.21.15.16:0/2076978466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f961804e440 con 0x7f962010de30 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 Processor -- start 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 -- start start 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ac4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dc49afdc0 con 0x562dc471e520 2024-09-21T02:13:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x562dc4a3c1f0 con 0x562dc46f5660 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f962d787740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562dc4a2b620 con 0x562dc46fd840 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.310+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36788/0 (socket says 172.21.15.16:36788) 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 -- 172.21.15.16:0/3441762853 learned_addr learned my addr 172.21.15.16:0/3441762853 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9624faf640 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ac4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9615ffb640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 0x7f96080b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f96257b0640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 0x7f96080b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 -- 172.21.15.16:0/3441762853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 msgr2=0x562dc4ac4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 -- 172.21.15.16:0/3441762853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 msgr2=0x562dc4ac4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ac4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 -- 172.21.15.16:0/3441762853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc49f3fd0 con 0x562dc471e520 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9615ffb640 1 -- 172.21.15.16:0/2076978466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f9618108cf0 con 0x7f962010de30 2024-09-21T02:13:12.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9617fff640 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9610018cb0 tx=0x7f961000af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3e716eca195621 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.311+0000 7f9606ffd640 1 -- 172.21.15.16:0/3441762853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610018e60 con 0x562dc471e520 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.312+0000 7f96257b0640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 0x7f96080b8b90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9618106180 tx=0x7f96180d2040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.312+0000 7f9606ffd640 1 -- 172.21.15.16:0/3441762853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9610016860 con 0x562dc471e520 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.312+0000 7f9606ffd640 1 -- 172.21.15.16:0/3441762853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610005240 con 0x562dc471e520 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.312+0000 7f962d787740 1 -- 172.21.15.16:0/3441762853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 msgr2=0x562dc4ac4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.312+0000 7f962d787740 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9610018cb0 tx=0x7f961000af90 comp rx=0 tx=0).stop 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 -- 172.21.15.16:0/3441762853 shutdown_connections 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ac4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 --2- 172.21.15.16:0/3441762853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4ac4510 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 -- 172.21.15.16:0/3441762853 >> 172.21.15.16:0/3441762853 conn(0x562dc4712910 msgr2=0x562dc4ac2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 -- 172.21.15.16:0/3441762853 shutdown_connections 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 -- 172.21.15.16:0/3441762853 wait complete. 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 -- start start 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.313+0000 7f962d787740 1 Processor -- start 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4acfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ad00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dc49c6300 con 0x562dc471e520 2024-09-21T02:13:12.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x562dc4a2f430 con 0x562dc46f5660 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562dc488cbb0 con 0x562dc46fd840 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f962d787740 10 client.0 _ll_register_callbacks cb 0x562dc49e0120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36792/0 (socket says 172.21.15.16:36792) 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 -- 172.21.15.16:0/939611069 learned_addr learned my addr 172.21.15.16:0/939611069 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9624faf640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ad00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4acfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 -- 172.21.15.16:0/939611069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 msgr2=0x562dc4ad00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ad00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 -- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 msgr2=0x562dc4acfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4acfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 -- 172.21.15.16:0/939611069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc49b5780 con 0x562dc471e520 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f9617fff640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f961000a6d0 tx=0x7f961000acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c2c6ca38077b58 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f95fffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610028070 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.314+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f9610016c20 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9610016860 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9610004020 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 10 client.15987 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562dc4789b00 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9610016860 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f9610018110 con 0x562dc471e520 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 10 client.15987 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 10 client.15987 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 20 client.15987 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46dcea0 0x562dc4ae1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 4 client.15987 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:13:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dc4a3e140 con 0x562dc46dcea0 2024-09-21T02:13:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.315+0000 7f9624faf640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46dcea0 0x562dc4ae1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.316+0000 7f95fffff640 10 client.15987 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.316+0000 7f9624faf640 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46dcea0 0x562dc4ae1e20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a75c7f0ab5de486 server_cookie=8f8e9e14fa20cfde in_seq=0 out_seq=0 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f95fffff640 1 -- 172.21.15.16:0/939611069 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x562dc4a3e140 con 0x562dc46dcea0 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f95fffff640 10 client.15987 handle_command_reply: tid=0 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 client.15987 shutdown 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 msgr2=0x562dc4adbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f961000a6d0 tx=0x7f961000acd0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 shutdown_connections 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46dcea0 0x562dc4ae1e20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc471e520 0x562dc4adbe90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ad00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f95fffff640 0 client.15987 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 --2- 172.21.15.16:0/939611069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4acfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 >> 172.21.15.16:0/939611069 conn(0x562dc4712910 msgr2=0x562dc4ac76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 shutdown_connections 2024-09-21T02:13:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 1 -- 172.21.15.16:0/939611069 wait complete. 2024-09-21T02:13:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.317+0000 7f962d787740 20 client.15987 trim_cache size 0 max 16384 2024-09-21T02:13:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.334+0000 7f962d787740 1 Processor -- start 2024-09-21T02:13:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.334+0000 7f962d787740 1 -- start start 2024-09-21T02:13:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc471e520 0x562dc4accd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96201aeb00 con 0x562dc46f5660 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96201b0640 con 0x562dc471e520 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f962d787740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96201af1d0 con 0x562dc46fd840 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36802/0 (socket says 172.21.15.16:36802) 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 -- 172.21.15.16:0/2042879914 learned_addr learned my addr 172.21.15.16:0/2042879914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9624faf640 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 -- 172.21.15.16:0/2042879914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 msgr2=0x562dc4ac5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 -- 172.21.15.16:0/2042879914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc471e520 msgr2=0x562dc4accd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc471e520 0x562dc4accd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 -- 172.21.15.16:0/2042879914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc4800960 con 0x562dc46f5660 2024-09-21T02:13:12.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9624faf640 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:13:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.335+0000 7f9617fff640 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f961000b010 tx=0x7f96100189d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336e712761d4a3f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f96067fc640 1 -- 172.21.15.16:0/2042879914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961003e070 con 0x562dc46f5660 2024-09-21T02:13:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f96067fc640 1 -- 172.21.15.16:0/2042879914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9610016860 con 0x562dc46f5660 2024-09-21T02:13:12.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f96067fc640 1 -- 172.21.15.16:0/2042879914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610028030 con 0x562dc46f5660 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 -- 172.21.15.16:0/2042879914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 msgr2=0x562dc4adbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f961000b010 tx=0x7f96100189d0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 -- 172.21.15.16:0/2042879914 shutdown_connections 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc471e520 0x562dc4accd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc46fd840 0x562dc4ac5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 --2- 172.21.15.16:0/2042879914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46f5660 0x562dc4adbd90 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 -- 172.21.15.16:0/2042879914 >> 172.21.15.16:0/2042879914 conn(0x562dc4712910 msgr2=0x562dc4ac5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 -- 172.21.15.16:0/2042879914 shutdown_connections 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.336+0000 7f962d787740 1 -- 172.21.15.16:0/2042879914 wait complete. 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 -- start start 2024-09-21T02:13:12.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 Processor -- start 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ad40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 0x562dc4b4b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96201aff70 con 0x562dc46fd840 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96201af8a0 con 0x562dc46f5660 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96201acb00 con 0x562dc471e520 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f962d787740 10 client.0 _ll_register_callbacks cb 0x7f961c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.337+0000 7f9624faf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36806/0 (socket says 172.21.15.16:36806) 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 -- 172.21.15.16:0/145338582 learned_addr learned my addr 172.21.15.16:0/145338582 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9617fff640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ad40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9617fff640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 0x562dc4b4b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 -- 172.21.15.16:0/145338582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 msgr2=0x562dc4b4b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 0x562dc4b4b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 -- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 msgr2=0x562dc4ad40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ad40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 -- 172.21.15.16:0/145338582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc4ad4e70 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9617fff640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 0x562dc4b4b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f9624faf640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x562dc4ae2390 tx=0x7f961c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cee3cb8260aaf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f95fffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c005020 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f961c0051c0 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.338+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9610016860 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f961c031050 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 10 client.15993 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562dc4b4cbb0 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9610016860 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f961c024470 con 0x562dc46fd840 2024-09-21T02:13:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 10 client.15993 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 10 client.15993 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 20 client.15993 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46b2250 0x562dc4b4f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 4 client.15993 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c96ba1ec", "release": true, "await": true}] 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c96ba1ec", "release": true, "await": true}) -- 0x562dc4a39bf0 con 0x562dc46b2250 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.339+0000 7f9617fff640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46b2250 0x562dc4b4f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.340+0000 7f9617fff640 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46b2250 0x562dc4b4f3f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4775438a297673c9 server_cookie=6a9bd2538dad3c3c in_seq=0 out_seq=0 2024-09-21T02:13:12.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.340+0000 7f95fffff640 10 client.15993 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.345+0000 7f95fffff640 1 -- 172.21.15.16:0/145338582 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562dc4a39bf0 con 0x562dc46b2250 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.345+0000 7f95fffff640 10 client.15993 handle_command_reply: tid=0 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.345+0000 7f962d787740 1 client.15993 shutdown 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 msgr2=0x562dc4ad45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x562dc4ae2390 tx=0x7f961c00bf90 comp rx=0 tx=0).stop 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 shutdown_connections 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x562dc46b2250 0x562dc4b4f3f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562dc471e520 0x562dc4b4b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dc46fd840 0x562dc4ad45f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 --2- 172.21.15.16:0/145338582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x562dc46f5660 0x562dc4ad40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 >> 172.21.15.16:0/145338582 conn(0x562dc4712910 msgr2=0x562dc4acedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f95fffff640 0 client.15993 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 shutdown_connections 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 1 -- 172.21.15.16:0/145338582 wait complete. 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.346+0000 7f962d787740 20 client.15993 trim_cache size 0 max 16384 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:13:12.348 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 24, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "c96ba1ec": { 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 24, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.2, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:13:12.349 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:13:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 msgr2=0x7f96080b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 0x7f96080b8b90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9618106180 tx=0x7f96180d2040 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 msgr2=0x7f962019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9618005cc0 tx=0x7f9618032a10 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 shutdown_connections 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f96080b6680 0x7f96080b8b90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f962010de30 0x7f962019fba0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9620105320 0x7f962019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 --2- 172.21.15.16:0/2076978466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96201049d0 0x7f962019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 >> 172.21.15.16:0/2076978466 conn(0x7f9620079740 msgr2=0x7f96201007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 shutdown_connections 2024-09-21T02:13:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:12.347+0000 7f9627213640 1 -- 172.21.15.16:0/2076978466 wait complete. 2024-09-21T02:13:13.200 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c96ba1ec', total seconds elapsed: 22.8 2024-09-21T02:13:13.200 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-09-21T02:13:13.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:13:13.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:13 smithi016 ceph-mon[31245]: pgmap v534: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 708 KiB/s wr, 35 op/s 2024-09-21T02:13:13.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:13 smithi161 ceph-mon[34061]: pgmap v534: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 708 KiB/s wr, 35 op/s 2024-09-21T02:13:13.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:13 smithi123 ceph-mon[33453]: pgmap v534: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 708 KiB/s wr, 35 op/s 2024-09-21T02:13:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:15.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:15 smithi016 ceph-mon[31245]: pgmap v535: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 69 op/s 2024-09-21T02:13:15.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:15 smithi161 ceph-mon[34061]: pgmap v535: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 69 op/s 2024-09-21T02:13:15.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:15 smithi123 ceph-mon[33453]: pgmap v535: 129 pgs: 129 active+clean; 15 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.7 MiB/s wr, 69 op/s 2024-09-21T02:13:17.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:17 smithi016 ceph-mon[31245]: pgmap v536: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 MiB/s wr, 90 op/s 2024-09-21T02:13:17.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:17 smithi161 ceph-mon[34061]: pgmap v536: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 MiB/s wr, 90 op/s 2024-09-21T02:13:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:17 smithi123 ceph-mon[33453]: pgmap v536: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 MiB/s wr, 90 op/s 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.882+0000 7f51523a8640 1 -- 172.21.15.16:0/2177296772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 msgr2=0x7f514c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.882+0000 7f51523a8640 1 --2- 172.21.15.16:0/2177296772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c1087b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5140009900 tx=0x7f514002c620 comp rx=0 tx=0).stop 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 -- 172.21.15.16:0/2177296772 shutdown_connections 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 --2- 172.21.15.16:0/2177296772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 --2- 172.21.15.16:0/2177296772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c1087b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 --2- 172.21.15.16:0/2177296772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 -- 172.21.15.16:0/2177296772 >> 172.21.15.16:0/2177296772 conn(0x7f514c079730 msgr2=0x7f514c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 -- 172.21.15.16:0/2177296772 shutdown_connections 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.883+0000 7f51523a8640 1 -- 172.21.15.16:0/2177296772 wait complete. 2024-09-21T02:13:18.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.884+0000 7f51523a8640 1 Processor -- start 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.884+0000 7f51523a8640 1 -- start start 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f514c111710 con 0x7f514c108cf0 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f514c111590 con 0x7f514c1083a0 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f51523a8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f514c111890 con 0x7f514c102a60 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f5150945640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f5150945640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47536/0 (socket says 172.21.15.16:47536) 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f5150945640 1 -- 172.21.15.16:0/784288041 learned_addr learned my addr 172.21.15.16:0/784288041 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514bfff640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514b7fe640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:18.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514bfff640 1 -- 172.21.15.16:0/784288041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 msgr2=0x7f514c198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514bfff640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514bfff640 1 -- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 msgr2=0x7f514c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.885+0000 7f514bfff640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f514bfff640 1 -- 172.21.15.16:0/784288041 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c19e230 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f5150945640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f514bfff640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c1982c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f513c00e9f0 tx=0x7f513c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00ccc0 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f513c005020 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c19e520 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00ce60 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.886+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1a4c80 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.888+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c102e70 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.889+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f513c004ae0 con 0x7f514c102a60 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.890+0000 7f51497fa640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 0x7f51340b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:18.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.890+0000 7f514b7fe640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 0x7f51340b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.890+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f513c0e7a50 con 0x7f514c102a60 2024-09-21T02:13:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.890+0000 7f514b7fe640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 0x7f51340b8b90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5140004770 tx=0x7f51400058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:18.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:17.894+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f513c0adc30 con 0x7f514c102a60 2024-09-21T02:13:18.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.318+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f514c10aac0 con 0x7f514c102a60 2024-09-21T02:13:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.322+0000 7f51497fa640 1 -- 172.21.15.16:0/784288041 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f513c0b2010 con 0x7f514c102a60 2024-09-21T02:13:18.323 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:13:18.323 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:13:18.323 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:13:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.326+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 msgr2=0x7f51340b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.326+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 0x7f51340b8b90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5140004770 tx=0x7f51400058d0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.326+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 msgr2=0x7f514c1982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:18.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.326+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c1982c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f513c00e9f0 tx=0x7f513c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.327+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 shutdown_connections 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.327+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f51340b6680 0x7f51340b8b90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.328+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f514c108cf0 0x7f514c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.328+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f514c1083a0 0x7f514c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.328+0000 7f51523a8640 1 --2- 172.21.15.16:0/784288041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c102a60 0x7f514c1982c0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f513c00e9f0 tx=0x7f513c00eeb0 comp rx=0 tx=0).stop 2024-09-21T02:13:18.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.328+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 >> 172.21.15.16:0/784288041 conn(0x7f514c079730 msgr2=0x7f514c1045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:18.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.330+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 shutdown_connections 2024-09-21T02:13:18.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:18.337+0000 7f51523a8640 1 -- 172.21.15.16:0/784288041 wait complete. 2024-09-21T02:13:18.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:18 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/784288041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:18.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:18 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/784288041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:18.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:18 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/784288041' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:19 smithi016 ceph-mon[31245]: pgmap v537: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.5 MiB/s wr, 145 op/s 2024-09-21T02:13:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:19 smithi161 ceph-mon[34061]: pgmap v537: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.5 MiB/s wr, 145 op/s 2024-09-21T02:13:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:19 smithi123 ceph-mon[33453]: pgmap v537: 129 pgs: 129 active+clean; 15 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.5 MiB/s wr, 145 op/s 2024-09-21T02:13:21.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:21 smithi016 ceph-mon[31245]: pgmap v538: 129 pgs: 129 active+clean; 15 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 15 MiB/s wr, 246 op/s 2024-09-21T02:13:21.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:21 smithi161 ceph-mon[34061]: pgmap v538: 129 pgs: 129 active+clean; 15 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 15 MiB/s wr, 246 op/s 2024-09-21T02:13:21.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:21 smithi123 ceph-mon[33453]: pgmap v538: 129 pgs: 129 active+clean; 15 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 15 MiB/s wr, 246 op/s 2024-09-21T02:13:22.525 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0' 2024-09-21T02:13:22.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 139.25 --await-for 45.0 2024-09-21T02:13:23.176 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:23.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:23 smithi016 ceph-mon[31245]: pgmap v539: 129 pgs: 129 active+clean; 15 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 330 op/s 2024-09-21T02:13:23.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:23 smithi161 ceph-mon[34061]: pgmap v539: 129 pgs: 129 active+clean; 15 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 330 op/s 2024-09-21T02:13:23.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:23 smithi123 ceph-mon[33453]: pgmap v539: 129 pgs: 129 active+clean; 15 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 20 MiB/s wr, 330 op/s 2024-09-21T02:13:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.943+0000 7ff657da2640 1 -- 172.21.15.16:0/4043418726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 msgr2=0x7ff6501098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.943+0000 7ff657da2640 1 --2- 172.21.15.16:0/4043418726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff6501098c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff648009ca0 tx=0x7ff64802a8f0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.944+0000 7ff657da2640 1 -- 172.21.15.16:0/4043418726 shutdown_connections 2024-09-21T02:13:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.944+0000 7ff657da2640 1 --2- 172.21.15.16:0/4043418726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.944+0000 7ff657da2640 1 --2- 172.21.15.16:0/4043418726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff6501098c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.944+0000 7ff657da2640 1 --2- 172.21.15.16:0/4043418726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff650103b70 0x7ff650103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.944+0000 7ff657da2640 1 -- 172.21.15.16:0/4043418726 >> 172.21.15.16:0/4043418726 conn(0x7ff6500ff5a0 msgr2=0x7ff6501019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.945+0000 7ff657da2640 1 -- 172.21.15.16:0/4043418726 shutdown_connections 2024-09-21T02:13:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.945+0000 7ff657da2640 1 -- 172.21.15.16:0/4043418726 wait complete. 2024-09-21T02:13:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.946+0000 7ff657da2640 1 Processor -- start 2024-09-21T02:13:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 -- start start 2024-09-21T02:13:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff650103b70 0x7ff65010a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff65010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff650115ba0 con 0x7ff650109e00 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff650115a20 con 0x7ff6501094b0 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.948+0000 7ff657da2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff650115d20 con 0x7ff650103b70 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.950+0000 7ff65633f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.950+0000 7ff65633f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47544/0 (socket says 172.21.15.16:47544) 2024-09-21T02:13:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.950+0000 7ff65633f640 1 -- 172.21.15.16:0/3640046722 learned_addr learned my addr 172.21.15.16:0/3640046722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.952+0000 7ff65633f640 1 -- 172.21.15.16:0/3640046722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff650103b70 msgr2=0x7ff65010a4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:13:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.952+0000 7ff65633f640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff650103b70 0x7ff65010a4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65533d640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff65010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65633f640 1 -- 172.21.15.16:0/3640046722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 msgr2=0x7ff65010aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:23.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65633f640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff65010aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65633f640 1 -- 172.21.15.16:0/3640046722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff650113a70 con 0x7ff650109e00 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65533d640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff65010aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.953+0000 7ff65633f640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff64000e970 tx=0x7ff64000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.954+0000 7ff646ffd640 1 -- 172.21.15.16:0/3640046722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64000ccc0 con 0x7ff650109e00 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.954+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff650113d60 con 0x7ff650109e00 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.954+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6501142a0 con 0x7ff650109e00 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.954+0000 7ff646ffd640 1 -- 172.21.15.16:0/3640046722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff64000ce60 con 0x7ff650109e00 2024-09-21T02:13:23.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.954+0000 7ff646ffd640 1 -- 172.21.15.16:0/3640046722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640010430 con 0x7ff650109e00 2024-09-21T02:13:23.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.959+0000 7ff65e316740 1 Processor -- start 2024-09-21T02:13:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.964+0000 7ff65e316740 1 -- start start 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 0x55869a2a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55869a21cbb0 con 0x558699edeb70 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55869a20bfe0 con 0x558699eff850 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65e316740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55869a20fdf0 con 0x558699ed6990 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65533d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 0x55869a2a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65533d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 0x55869a2a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47548/0 (socket says 172.21.15.16:47548) 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65533d640 1 -- 172.21.15.16:0/2313045789 learned_addr learned my addr 172.21.15.16:0/2313045789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.965+0000 7ff65533d640 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.966+0000 7ff646ffd640 1 -- 172.21.15.16:0/3640046722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff6400046c0 con 0x7ff650109e00 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff646ffd640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 0x7ff63c0b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 -- 172.21.15.16:0/2313045789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 msgr2=0x55869a2a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 -- 172.21.15.16:0/2313045789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 msgr2=0x55869a2a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 0x55869a2a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.968+0000 7ff655b3e640 1 -- 172.21.15.16:0/2313045789 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558699f79ba0 con 0x558699ed6990 2024-09-21T02:13:23.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.969+0000 7ff655b3e640 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2a63b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009aa0 tx=0x7ff64c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5da3f93ef9f2afdc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:23.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.969+0000 7ff646ffd640 1 -- 172.21.15.16:0/3640046722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff6400e7db0 con 0x7ff650109e00 2024-09-21T02:13:23.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.971+0000 7ff63bfff640 1 -- 172.21.15.16:0/2313045789 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c00ece0 con 0x558699ed6990 2024-09-21T02:13:23.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.971+0000 7ff63bfff640 1 -- 172.21.15.16:0/2313045789 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff64c00ee80 con 0x558699ed6990 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.972+0000 7ff63bfff640 1 -- 172.21.15.16:0/2313045789 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c004a00 con 0x558699ed6990 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.972+0000 7ff65e316740 1 -- 172.21.15.16:0/2313045789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 msgr2=0x55869a2a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.972+0000 7ff65e316740 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2a63b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009aa0 tx=0x7ff64c009f60 comp rx=0 tx=0).stop 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65633f640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 0x7ff63c0b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65e316740 1 -- 172.21.15.16:0/2313045789 shutdown_connections 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65e316740 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699edeb70 0x55869a2a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65e316740 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2a63b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65e316740 1 --2- 172.21.15.16:0/2313045789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.975+0000 7ff65e316740 1 -- 172.21.15.16:0/2313045789 >> 172.21.15.16:0/2313045789 conn(0x558699ef3c40 msgr2=0x55869a2a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.976+0000 7ff65633f640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 0x7ff63c0b8cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff640020000 tx=0x7ff6400a9040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.984+0000 7ff65e316740 1 -- 172.21.15.16:0/2313045789 shutdown_connections 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.984+0000 7ff65e316740 1 -- 172.21.15.16:0/2313045789 wait complete. 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.985+0000 7ff65e316740 1 -- start start 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.985+0000 7ff65e316740 1 Processor -- start 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699eff850 0x55869a2bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55869a06d7b0 con 0x558699eff850 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55869a1a6e90 con 0x558699edeb70 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55869a2b46a0 con 0x558699ed6990 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65e316740 10 client.0 _ll_register_callbacks cb 0x55869a209710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65533d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65533d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:41516/0 (socket says 172.21.15.16:41516) 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.986+0000 7ff65533d640 1 -- 172.21.15.16:0/519389944 learned_addr learned my addr 172.21.15.16:0/519389944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 -- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 msgr2=0x55869a2b1f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b1f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 -- 172.21.15.16:0/519389944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699eff850 msgr2=0x55869a2bda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699eff850 0x55869a2bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 -- 172.21.15.16:0/519389944 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869a2bdfb0 con 0x558699ed6990 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.987+0000 7ff65533d640 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff648030fd0 tx=0x7ff648032fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc3dbf15c2a4f0c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.988+0000 7ff6397fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.988+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648003b70 con 0x558699ed6990 2024-09-21T02:13:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.988+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55869a21eb00 con 0x558699ed6990 2024-09-21T02:13:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.988+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff648003d10 con 0x558699ed6990 2024-09-21T02:13:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.988+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff648004050 con 0x558699ed6990 2024-09-21T02:13:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.990+0000 7ff65e316740 10 client.25675 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.990+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55869a2c08e0 con 0x558699ed6990 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.990+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55869a21eb00 con 0x558699ed6990 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.991+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7ff648047070 con 0x558699ed6990 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 10 client.25675 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 10 client.25675 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 20 client.25675 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699ebe1d0 0x55869a2c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 4 client.25675 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55869a21a5b0 con 0x558699ebe1d0 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.992+0000 7ff655b3e640 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699ebe1d0 0x55869a2c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.993+0000 7ff655b3e640 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699ebe1d0 0x55869a2c4240 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fae4deada422914 server_cookie=fb6eeb5c70843248 in_seq=0 out_seq=0 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.993+0000 7ff6397fa640 10 client.25675 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.994+0000 7ff6397fa640 1 -- 172.21.15.16:0/519389944 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55869a21a5b0 con 0x558699ebe1d0 2024-09-21T02:13:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.994+0000 7ff6397fa640 10 client.25675 handle_command_reply: tid=0 2024-09-21T02:13:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:23.994+0000 7ff65e316740 1 client.25675 shutdown 2024-09-21T02:13:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.002+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 msgr2=0x55869a2b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.002+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff648030fd0 tx=0x7ff648032fb0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 shutdown_connections 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699ebe1d0 0x55869a2c4240 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699eff850 0x55869a2bda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 --2- 172.21.15.16:0/519389944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699ed6990 0x55869a2b19d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 >> 172.21.15.16:0/519389944 conn(0x558699ef3c40 msgr2=0x55869a2a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff6397fa640 0 client.25675 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 shutdown_connections 2024-09-21T02:13:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 1 -- 172.21.15.16:0/519389944 wait complete. 2024-09-21T02:13:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.005+0000 7ff65e316740 20 client.25675 trim_cache size 0 max 16384 2024-09-21T02:13:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.041+0000 7ff65e316740 1 Processor -- start 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.041+0000 7ff65e316740 1 -- start start 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65e316740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2bd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65e316740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699edeb70 0x55869a2b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65e316740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65e316740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff63400fe00 con 0x558699ed6990 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65e316740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff634011940 con 0x558699eff850 2024-09-21T02:13:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65533d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65533d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:44542/0 (socket says 172.21.15.16:44542) 2024-09-21T02:13:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.042+0000 7ff65533d640 1 -- 172.21.15.16:0/154133581 learned_addr learned my addr 172.21.15.16:0/154133581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6340104d0 con 0x558699edeb70 2024-09-21T02:13:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2bd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 -- 172.21.15.16:0/154133581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699edeb70 msgr2=0x55869a2b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699edeb70 0x55869a2b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 -- 172.21.15.16:0/154133581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 msgr2=0x55869a2bd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2bd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.043+0000 7ff65533d640 1 -- 172.21.15.16:0/154133581 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869a2a3cc0 con 0x558699eff850 2024-09-21T02:13:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.044+0000 7ff65533d640 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6480098a0 tx=0x7ff64803e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d5983668a4faf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.046+0000 7ff63b7fe640 1 -- 172.21.15.16:0/154133581 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480319c0 con 0x558699eff850 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.046+0000 7ff63b7fe640 1 -- 172.21.15.16:0/154133581 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff648036030 con 0x558699eff850 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.046+0000 7ff63b7fe640 1 -- 172.21.15.16:0/154133581 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648004430 con 0x558699eff850 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 msgr2=0x55869a2b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6480098a0 tx=0x7ff64803e040 comp rx=0 tx=0).stop 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 shutdown_connections 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699eff850 0x55869a2b0590 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699edeb70 0x55869a2b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 --2- 172.21.15.16:0/154133581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2bd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.047+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 >> 172.21.15.16:0/154133581 conn(0x558699ef3c40 msgr2=0x55869a2ac910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.049+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 shutdown_connections 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.049+0000 7ff65e316740 1 -- 172.21.15.16:0/154133581 wait complete. 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.050+0000 7ff65e316740 1 -- start start 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.050+0000 7ff65e316740 1 Processor -- start 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2b9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff634011270 con 0x558699ed6990 2024-09-21T02:13:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff634010ba0 con 0x558699edeb70 2024-09-21T02:13:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff63400de90 con 0x558699eff850 2024-09-21T02:13:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.051+0000 7ff65e316740 10 client.0 _ll_register_callbacks cb 0x7ff64c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.054+0000 7ff65533d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.054+0000 7ff65533d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:41532/0 (socket says 172.21.15.16:41532) 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.054+0000 7ff65533d640 1 -- 172.21.15.16:0/3433713233 learned_addr learned my addr 172.21.15.16:0/3433713233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.054+0000 7ff65533d640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2b9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff655b3e640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 -- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 msgr2=0x55869a2b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 -- 172.21.15.16:0/3433713233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 msgr2=0x55869a2b9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2b9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 -- 172.21.15.16:0/3433713233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869a2b5f80 con 0x558699eff850 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.055+0000 7ff65533d640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55869a2b1760 tx=0x7ff648030c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92c4621d6afbd71e server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.056+0000 7ff6397fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:13:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.056+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648004b40 con 0x558699eff850 2024-09-21T02:13:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.056+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff648036030 con 0x558699eff850 2024-09-21T02:13:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.056+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7ff648004ce0 con 0x558699eff850 2024-09-21T02:13:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.056+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff6480041f0 con 0x558699eff850 2024-09-21T02:13:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.057+0000 7ff65e316740 10 client.25678 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.057+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55869a32e970 con 0x558699eff850 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff648036030 con 0x558699eff850 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7ff648066070 con 0x558699eff850 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 10 client.25678 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 10 client.25678 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 20 client.25678 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699e93580 0x55869a330e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 4 client.25678 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}] 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.058+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.25, "await_for": 45.0}) -- 0x55869a2179e0 con 0x558699e93580 2024-09-21T02:13:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.059+0000 7ff655b3e640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699e93580 0x55869a330e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.059+0000 7ff655b3e640 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699e93580 0x55869a330e40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a9b123eeb55680 server_cookie=a8355ab36cd8cc1e in_seq=0 out_seq=0 2024-09-21T02:13:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.060+0000 7ff6397fa640 10 client.25678 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.127+0000 7ff6397fa640 1 -- 172.21.15.16:0/3433713233 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55869a2179e0 con 0x558699e93580 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.127+0000 7ff6397fa640 10 client.25678 handle_command_reply: tid=0 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.127+0000 7ff65e316740 1 client.25678 shutdown 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 msgr2=0x55869a2b59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55869a2b1760 tx=0x7ff648030c90 comp rx=0 tx=0).stop 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 shutdown_connections 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558699e93580 0x55869a330e40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558699eff850 0x55869a2b59c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558699edeb70 0x55869a2b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 --2- 172.21.15.16:0/3433713233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558699ed6990 0x55869a2b9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 >> 172.21.15.16:0/3433713233 conn(0x558699ef3c40 msgr2=0x55869a2a8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff6397fa640 0 client.25678 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 shutdown_connections 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 1 -- 172.21.15.16:0/3433713233 wait complete. 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.128+0000 7ff65e316740 20 client.25678 trim_cache size 0 max 16384 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:13:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 26, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "1ea14e08": { 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 26, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.2, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:13:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 msgr2=0x7ff63c0b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 0x7ff63c0b8cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff640020000 tx=0x7ff6400a9040 comp rx=0 tx=0).stop 2024-09-21T02:13:24.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 msgr2=0x7ff650113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff64000e970 tx=0x7ff64000ed70 comp rx=0 tx=0).stop 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 shutdown_connections 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff63c0b67b0 0x7ff63c0b8cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff650109e00 0x7ff650113530 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6501094b0 0x7ff65010aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 --2- 172.21.15.16:0/3640046722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff650103b70 0x7ff65010a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 >> 172.21.15.16:0/3640046722 conn(0x7ff6500ff5a0 msgr2=0x7ff650106aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 shutdown_connections 2024-09-21T02:13:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:24.129+0000 7ff657da2640 1 -- 172.21.15.16:0/3640046722 wait complete. 2024-09-21T02:13:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:25 smithi016 ceph-mon[31245]: pgmap v540: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 28 MiB/s wr, 463 op/s 2024-09-21T02:13:25.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:25 smithi161 ceph-mon[34061]: pgmap v540: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 28 MiB/s wr, 463 op/s 2024-09-21T02:13:25.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:25 smithi123 ceph-mon[33453]: pgmap v540: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 28 MiB/s wr, 463 op/s 2024-09-21T02:13:26.349 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1ea14e08', quiesce took 13.1 seconds. Will release after: 6.15 2024-09-21T02:13:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:27 smithi016 ceph-mon[31245]: pgmap v541: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 28 MiB/s wr, 450 op/s 2024-09-21T02:13:27.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:27 smithi161 ceph-mon[34061]: pgmap v541: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 28 MiB/s wr, 450 op/s 2024-09-21T02:13:27.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:27 smithi123 ceph-mon[33453]: pgmap v541: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 28 MiB/s wr, 450 op/s 2024-09-21T02:13:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:29 smithi016 ceph-mon[31245]: pgmap v542: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-09-21T02:13:29.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:29 smithi161 ceph-mon[34061]: pgmap v542: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-09-21T02:13:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:29 smithi123 ceph-mon[33453]: pgmap v542: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 28 MiB/s wr, 451 op/s 2024-09-21T02:13:31.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:31 smithi161 ceph-mon[34061]: pgmap v543: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 404 op/s 2024-09-21T02:13:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:31 smithi123 ceph-mon[33453]: pgmap v543: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 404 op/s 2024-09-21T02:13:31.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:31 smithi016 ceph-mon[31245]: pgmap v543: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 404 op/s 2024-09-21T02:13:32.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:13:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 -- 172.21.15.16:0/3939001907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e601083a0 msgr2=0x7f3e601087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 --2- 172.21.15.16:0/3939001907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e601083a0 0x7f3e601087b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009900 tx=0x7f3e5402a410 comp rx=0 tx=0).stop 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 -- 172.21.15.16:0/3939001907 shutdown_connections 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 --2- 172.21.15.16:0/3939001907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e60108cf0 0x7f3e6010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 --2- 172.21.15.16:0/3939001907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e601083a0 0x7f3e601087b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 --2- 172.21.15.16:0/3939001907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 0x7f3e60102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.275+0000 7f3e668e9640 1 -- 172.21.15.16:0/3939001907 >> 172.21.15.16:0/3939001907 conn(0x7f3e60079730 msgr2=0x7f3e60079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.276+0000 7f3e668e9640 1 -- 172.21.15.16:0/3939001907 shutdown_connections 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.276+0000 7f3e668e9640 1 -- 172.21.15.16:0/3939001907 wait complete. 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.276+0000 7f3e668e9640 1 Processor -- start 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.276+0000 7f3e668e9640 1 -- start start 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 0x7f3e6019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:33.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e601083a0 0x7f3e6019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60112810 con 0x7f3e60108cf0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60112690 con 0x7f3e60102a60 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e668e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60112990 con 0x7f3e601083a0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50034/0 (socket says 172.21.15.16:50034) 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 -- 172.21.15.16:0/92694027 learned_addr learned my addr 172.21.15.16:0/92694027 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e5ffff640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 0x7f3e6019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e5f7fe640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e601083a0 0x7f3e6019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 -- 172.21.15.16:0/92694027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e601083a0 msgr2=0x7f3e6019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e601083a0 0x7f3e6019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 -- 172.21.15.16:0/92694027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 msgr2=0x7f3e6019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 0x7f3e6019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 -- 172.21.15.16:0/92694027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e601a0380 con 0x7f3e60108cf0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.277+0000 7f3e64e86640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800eaa0 tx=0x7f3e5800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:33.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.278+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5800cdb0 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.278+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e58005020 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.278+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e601a0670 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.278+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580116d0 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.278+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e601a6dd0 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.279+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e60103ce0 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.280+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3e580051d0 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.281+0000 7f3e5d7fa640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 0x7f3e480b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.281+0000 7f3e5ffff640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 0x7f3e480b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.281+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f3e580e7b40 con 0x7f3e60108cf0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.282+0000 7f3e5ffff640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 0x7f3e480b8c10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009bd0 tx=0x7f3e50005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:33.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.286+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3e58016030 con 0x7f3e60108cf0 2024-09-21T02:13:33.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.515+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e6019b540 con 0x7f3e60108cf0 2024-09-21T02:13:33.516 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:33 smithi016 ceph-mon[31245]: pgmap v544: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 308 op/s 2024-09-21T02:13:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.516+0000 7f3e5d7fa640 1 -- 172.21.15.16:0/92694027 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f3e580add20 con 0x7f3e60108cf0 2024-09-21T02:13:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:13:33.517 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:13:33.517 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:13:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 msgr2=0x7f3e480b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 0x7f3e480b8c10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009bd0 tx=0x7f3e50005f90 comp rx=0 tx=0).stop 2024-09-21T02:13:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 msgr2=0x7f3e6019fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800eaa0 tx=0x7f3e5800ef60 comp rx=0 tx=0).stop 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 shutdown_connections 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e480b6700 0x7f3e480b8c10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e60108cf0 0x7f3e6019fc50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e601083a0 0x7f3e6019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 --2- 172.21.15.16:0/92694027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e60102a60 0x7f3e6019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.518+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 >> 172.21.15.16:0/92694027 conn(0x7f3e60079730 msgr2=0x7f3e6006c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.519+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 shutdown_connections 2024-09-21T02:13:33.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:33.519+0000 7f3e668e9640 1 -- 172.21.15.16:0/92694027 wait complete. 2024-09-21T02:13:33.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:33 smithi161 ceph-mon[34061]: pgmap v544: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 308 op/s 2024-09-21T02:13:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:33 smithi123 ceph-mon[33453]: pgmap v544: 129 pgs: 129 active+clean; 15 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 19 MiB/s wr, 308 op/s 2024-09-21T02:13:34.257 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 1ea14e08 --release --await' 2024-09-21T02:13:34.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 1ea14e08 --release --await 2024-09-21T02:13:34.516 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:13:34.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:34 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/92694027' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:34.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:34 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/92694027' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:34.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:34 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/92694027' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.032+0000 7f71c21eb640 1 -- 172.21.15.16:0/197488456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108450 msgr2=0x7f71bc108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.032+0000 7f71c21eb640 1 --2- 172.21.15.16:0/197488456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108450 0x7f71bc108860 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009900 tx=0x7f71b002a410 comp rx=0 tx=0).stop 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 -- 172.21.15.16:0/197488456 shutdown_connections 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 --2- 172.21.15.16:0/197488456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108da0 0x7f71bc10f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 --2- 172.21.15.16:0/197488456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108450 0x7f71bc108860 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 --2- 172.21.15.16:0/197488456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 0x7f71bc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 -- 172.21.15.16:0/197488456 >> 172.21.15.16:0/197488456 conn(0x7f71bc0796d0 msgr2=0x7f71bc079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 -- 172.21.15.16:0/197488456 shutdown_connections 2024-09-21T02:13:35.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 -- 172.21.15.16:0/197488456 wait complete. 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.033+0000 7f71c21eb640 1 Processor -- start 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 -- start start 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 0x7f71bc19b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108450 0x7f71bc19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc112960 con 0x7f71bc108da0 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1127e0 con 0x7f71bc102b10 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71c21eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc112ae0 con 0x7f71bc108450 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50044/0 (socket says 172.21.15.16:50044) 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 -- 172.21.15.16:0/1963807225 learned_addr learned my addr 172.21.15.16:0/1963807225 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:35.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 0x7f71bc19b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71baffd640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108450 0x7f71bc19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 -- 172.21.15.16:0/1963807225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108450 msgr2=0x7f71bc19b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108450 0x7f71bc19b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 -- 172.21.15.16:0/1963807225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 msgr2=0x7f71bc19b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 0x7f71bc19b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.034+0000 7f71bbfff640 1 -- 172.21.15.16:0/1963807225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71bc1a65a0 con 0x7f71bc108da0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71bbfff640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f71b400eb10 tx=0x7f71b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71b8ff9640 1 -- 172.21.15.16:0/1963807225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4004b20 con 0x7f71bc108da0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71bc1a6890 con 0x7f71bc108da0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71b8ff9640 1 -- 172.21.15.16:0/1963807225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71b4004cc0 con 0x7f71bc108da0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71b8ff9640 1 -- 172.21.15.16:0/1963807225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4010690 con 0x7f71bc108da0 2024-09-21T02:13:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.035+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71bc1a6da0 con 0x7f71bc108da0 2024-09-21T02:13:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.038+0000 7f71c875f740 1 Processor -- start 2024-09-21T02:13:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71b8ff9640 1 -- 172.21.15.16:0/1963807225 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f71b4005000 con 0x7f71bc108da0 2024-09-21T02:13:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 -- start start 2024-09-21T02:13:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a85b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a85ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a85bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a746dc0 con 0x55dc5a494840 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a7d31f0 con 0x55dc5a4b5520 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71c875f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a7c2620 con 0x55dc5a48c660 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a85bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a85ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a85bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50060/0 (socket says 172.21.15.16:50060) 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.039+0000 7f71baffd640 1 -- 172.21.15.16:0/3781530996 learned_addr learned my addr 172.21.15.16:0/3781530996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71baffd640 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a85b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 -- 172.21.15.16:0/3781530996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 msgr2=0x55dc5a85b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71b8ff9640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 0x7f71a40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a85b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 -- 172.21.15.16:0/3781530996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 msgr2=0x55dc5a85bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a85bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 -- 172.21.15.16:0/3781530996 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc5a78afd0 con 0x55dc5a48c660 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71baffd640 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a85b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bbfff640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 0x7f71a40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a85ba50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bbc0 tx=0x7f71ac0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c440d909e9d8df9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71b8ff9640 1 -- 172.21.15.16:0/1963807225 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f71b40e8090 con 0x7f71bc108da0 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71a1ffb640 1 -- 172.21.15.16:0/3781530996 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac004030 con 0x55dc5a48c660 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71a1ffb640 1 -- 172.21.15.16:0/3781530996 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71ac0035f0 con 0x55dc5a48c660 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.040+0000 7f71a1ffb640 1 -- 172.21.15.16:0/3781530996 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac0116a0 con 0x55dc5a48c660 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 -- 172.21.15.16:0/3781530996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 msgr2=0x55dc5a85ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71bbfff640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 0x7f71a40b8bc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71b40e5a70 tx=0x7f71b400cf70 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a85ba50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bbc0 tx=0x7f71ac0107a0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 -- 172.21.15.16:0/3781530996 shutdown_connections 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a85bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a85ba50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 --2- 172.21.15.16:0/3781530996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a85b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 -- 172.21.15.16:0/3781530996 >> 172.21.15.16:0/3781530996 conn(0x55dc5a4a9910 msgr2=0x55dc5a859400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 -- 172.21.15.16:0/3781530996 shutdown_connections 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.041+0000 7f71c875f740 1 -- 172.21.15.16:0/3781530996 wait complete. 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 -- start start 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 Processor -- start 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a867140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a872e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a75d300 con 0x55dc5a48c660 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a7c6430 con 0x55dc5a494840 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.042+0000 7f71c875f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55dc5a623bb0 con 0x55dc5a4b5520 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71c875f740 10 client.0 _ll_register_callbacks cb 0x55dc5a777120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50076/0 (socket says 172.21.15.16:50076) 2024-09-21T02:13:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 -- 172.21.15.16:0/716932371 learned_addr learned my addr 172.21.15.16:0/716932371 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a867140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a872e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 -- 172.21.15.16:0/716932371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 msgr2=0x55dc5a872e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a872e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 -- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 msgr2=0x55dc5a867140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a867140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 -- 172.21.15.16:0/716932371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc5a74c780 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71baffd640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b0040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b687e9f0527a16 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f719affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0035040 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f71b003d450 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.043+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac0035f0 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71b0052050 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 10 client.16032 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc5a520b00 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71ac0035f0 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f71b003a040 con 0x55dc5a48c660 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 10 client.16032 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 10 client.16032 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 20 client.16032 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a473ea0 0x55dc5a878ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 4 client.16032 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc5a7d5140 con 0x55dc5a473ea0 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.044+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a473ea0 0x55dc5a878ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.045+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a473ea0 0x55dc5a878ef0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3234022ea02723bc server_cookie=567312ffdffddd46 in_seq=0 out_seq=0 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.045+0000 7f719affd640 10 client.16032 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f719affd640 1 -- 172.21.15.16:0/716932371 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dc5a7d5140 con 0x55dc5a473ea0 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f719affd640 10 client.16032 handle_command_reply: tid=0 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 client.16032 shutdown 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 msgr2=0x55dc5a866c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b0040040 comp rx=0 tx=0).stop 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 shutdown_connections 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a473ea0 0x55dc5a878ef0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a872e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a867140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f719affd640 0 client.16032 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 --2- 172.21.15.16:0/716932371 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a866c00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 >> 172.21.15.16:0/716932371 conn(0x55dc5a4a9910 msgr2=0x55dc5a85e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 shutdown_connections 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 1 -- 172.21.15.16:0/716932371 wait complete. 2024-09-21T02:13:35.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.046+0000 7f71c875f740 20 client.16032 trim_cache size 0 max 16384 2024-09-21T02:13:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 Processor -- start 2024-09-21T02:13:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 -- start start 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a872d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a865b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b0db0 con 0x55dc5a48c660 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b28f0 con 0x55dc5a494840 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71c875f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b1480 con 0x55dc5a4b5520 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50086/0 (socket says 172.21.15.16:50086) 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 -- 172.21.15.16:0/914610658 learned_addr learned my addr 172.21.15.16:0/914610658 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 -- 172.21.15.16:0/914610658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 msgr2=0x55dc5a865b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a865b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 -- 172.21.15.16:0/914610658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 msgr2=0x55dc5a872d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a872d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 -- 172.21.15.16:0/914610658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc5a868fd0 con 0x55dc5a48c660 2024-09-21T02:13:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.064+0000 7f71baffd640 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f71b00092a0 tx=0x7f71b0009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8c14f0a536c52b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71a17fa640 1 -- 172.21.15.16:0/914610658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0035070 con 0x55dc5a48c660 2024-09-21T02:13:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71a17fa640 1 -- 172.21.15.16:0/914610658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71b004c0e0 con 0x55dc5a48c660 2024-09-21T02:13:35.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71a17fa640 1 -- 172.21.15.16:0/914610658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00052a0 con 0x55dc5a48c660 2024-09-21T02:13:35.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 -- 172.21.15.16:0/914610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 msgr2=0x55dc5a85ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f71b00092a0 tx=0x7f71b0009830 comp rx=0 tx=0).stop 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 -- 172.21.15.16:0/914610658 shutdown_connections 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a4b5520 0x55dc5a865b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a494840 0x55dc5a872d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 --2- 172.21.15.16:0/914610658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a48c660 0x55dc5a85ceb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 -- 172.21.15.16:0/914610658 >> 172.21.15.16:0/914610658 conn(0x55dc5a4a9910 msgr2=0x55dc5a865480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 -- 172.21.15.16:0/914610658 shutdown_connections 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.065+0000 7f71c875f740 1 -- 172.21.15.16:0/914610658 wait complete. 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 -- start start 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 Processor -- start 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a86f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a86bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b2220 con 0x55dc5a494840 2024-09-21T02:13:35.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b1b50 con 0x55dc5a4b5520 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1aed90 con 0x55dc5a48c660 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.066+0000 7f71c875f740 10 client.0 _ll_register_callbacks cb 0x7f71b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50100/0 (socket says 172.21.15.16:50100) 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 -- 172.21.15.16:0/2979567886 learned_addr learned my addr 172.21.15.16:0/2979567886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71baffd640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a86bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71baffd640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a86f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 -- 172.21.15.16:0/2979567886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 msgr2=0x55dc5a86f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a86f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 -- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 msgr2=0x55dc5a86bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a86bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 -- 172.21.15.16:0/2979567886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc5a597960 con 0x55dc5a494840 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71bb7fe640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x55dc5a85a190 tx=0x7f71ac00bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9938d30c5fbb29 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f719affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac005240 con 0x55dc5a494840 2024-09-21T02:13:35.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9394+0+0 (secure 0 0 0) 0x7f71ac0053e0 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b004c0e0 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.067+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71ac034050 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 10 client.16038 fetch_fsmap learned FSMap version 28 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc5a8e4620 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71b004c0e0 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f71ac011560 con 0x55dc5a494840 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 10 client.16038 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 10 client.16038 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 20 client.16038 populate_metadata read hostname 'smithi016' 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a449250 0x55dc5a8e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 4 client.16038 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1ea14e08", "release": true, "await": true}] 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1ea14e08", "release": true, "await": true}) -- 0x55dc5a7d0bf0 con 0x55dc5a449250 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.068+0000 7f71baffd640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a449250 0x55dc5a8e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.069+0000 7f71baffd640 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a449250 0x55dc5a8e6a70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b504d7724da8a578 server_cookie=3010d034ff6802ec in_seq=0 out_seq=0 2024-09-21T02:13:35.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.069+0000 7f719affd640 10 client.16038 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f719affd640 1 -- 172.21.15.16:0/2979567886 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55dc5a7d0bf0 con 0x55dc5a449250 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f719affd640 10 client.16038 handle_command_reply: tid=0 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 client.16038 shutdown 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 msgr2=0x55dc5a86ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x55dc5a85a190 tx=0x7f71ac00bfb0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 shutdown_connections 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55dc5a449250 0x55dc5a8e6a70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dc5a4b5520 0x55dc5a86bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55dc5a494840 0x55dc5a86ba10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f719affd640 0 client.16038 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 --2- 172.21.15.16:0/2979567886 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55dc5a48c660 0x55dc5a86f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.074+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 >> 172.21.15.16:0/2979567886 conn(0x55dc5a4a9910 msgr2=0x55dc5a865e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.075+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 shutdown_connections 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.075+0000 7f71c875f740 1 -- 172.21.15.16:0/2979567886 wait complete. 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.075+0000 7f71c875f740 20 client.16038 trim_cache size 0 max 16384 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 28, 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "1ea14e08": { 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 28, 2024-09-21T02:13:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 139.2, 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:13:35.079 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:13:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 msgr2=0x7f71a40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 0x7f71a40b8bc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71b40e5a70 tx=0x7f71b400cf70 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 msgr2=0x7f71bc19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f71b400eb10 tx=0x7f71b400c2d0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 shutdown_connections 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f71a40b66b0 0x7f71a40b8bc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71bc108da0 0x7f71bc19bd90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71bc108450 0x7f71bc19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 --2- 172.21.15.16:0/1963807225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71bc102b10 0x7f71bc19b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 >> 172.21.15.16:0/1963807225 conn(0x7f71bc0796d0 msgr2=0x7f71bc106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 shutdown_connections 2024-09-21T02:13:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:13:35.076+0000 7f71c21eb640 1 -- 172.21.15.16:0/1963807225 wait complete. 2024-09-21T02:13:35.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:35 smithi016 ceph-mon[31245]: pgmap v545: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 231 op/s 2024-09-21T02:13:35.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:35 smithi123 ceph-mon[33453]: pgmap v545: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 231 op/s 2024-09-21T02:13:35.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:35 smithi161 ceph-mon[34061]: pgmap v545: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 231 op/s 2024-09-21T02:13:36.055 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1ea14e08', total seconds elapsed: 22.9 2024-09-21T02:13:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:37 smithi123 ceph-mon[33453]: pgmap v546: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 104 op/s 2024-09-21T02:13:37.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:37 smithi016 ceph-mon[31245]: pgmap v546: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 104 op/s 2024-09-21T02:13:37.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:37 smithi161 ceph-mon[34061]: pgmap v546: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 104 op/s 2024-09-21T02:13:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:39 smithi123 ceph-mon[33453]: pgmap v547: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 117 op/s 2024-09-21T02:13:39.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:39 smithi016 ceph-mon[31245]: pgmap v547: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 117 op/s 2024-09-21T02:13:39.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:39 smithi161 ceph-mon[34061]: pgmap v547: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 117 op/s 2024-09-21T02:13:41.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:41 smithi123 ceph-mon[33453]: pgmap v548: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 176 op/s 2024-09-21T02:13:41.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:41 smithi016 ceph-mon[31245]: pgmap v548: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 176 op/s 2024-09-21T02:13:41.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:41 smithi161 ceph-mon[34061]: pgmap v548: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 176 op/s 2024-09-21T02:13:43.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:43 smithi016 ceph-mon[31245]: pgmap v549: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T02:13:43.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:43 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:13:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:43 smithi123 ceph-mon[33453]: pgmap v549: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T02:13:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:43 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:13:43.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:43 smithi161 ceph-mon[34061]: pgmap v549: 129 pgs: 129 active+clean; 16 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 287 op/s 2024-09-21T02:13:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:43 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: Unable to set osd_memory_target on smithi123 to 184577433: error parsing value: Value '184577433' is below minimum 939524096 2024-09-21T02:13:44.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: Unable to set osd_memory_target on smithi161 to 184581734: error parsing value: Value '184581734' is below minimum 939524096 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:13:44.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:13:44.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: Unable to set osd_memory_target on smithi123 to 184577433: error parsing value: Value '184577433' is below minimum 939524096 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: Unable to set osd_memory_target on smithi161 to 184581734: error parsing value: Value '184581734' is below minimum 939524096 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:13:44.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:13:44.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi123 to 176.0M 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi161 to 176.0M 2024-09-21T02:13:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: Unable to set osd_memory_target on smithi123 to 184577433: error parsing value: Value '184577433' is below minimum 939524096 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: Unable to set osd_memory_target on smithi161 to 184581734: error parsing value: Value '184581734' is below minimum 939524096 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:13:44.916 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:13:45.739 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:45 smithi123 ceph-mon[33453]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-21T02:13:45.740 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:45 smithi123 ceph-mon[33453]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-21T02:13:45.740 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:45 smithi123 ceph-mon[33453]: pgmap v550: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 459 op/s 2024-09-21T02:13:45.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:45 smithi016 ceph-mon[31245]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-21T02:13:45.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:45 smithi016 ceph-mon[31245]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-21T02:13:45.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:45 smithi016 ceph-mon[31245]: pgmap v550: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 459 op/s 2024-09-21T02:13:45.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:45 smithi161 ceph-mon[34061]: Adjusting osd_memory_target on smithi016 to 177.0M 2024-09-21T02:13:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:45 smithi161 ceph-mon[34061]: Unable to set osd_memory_target on smithi016 to 185688473: error parsing value: Value '185688473' is below minimum 939524096 2024-09-21T02:13:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:45 smithi161 ceph-mon[34061]: pgmap v550: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 28 MiB/s wr, 459 op/s 2024-09-21T02:13:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:47 smithi123 ceph-mon[33453]: pgmap v551: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 494 op/s 2024-09-21T02:13:47.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:47 smithi016 ceph-mon[31245]: pgmap v551: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 494 op/s 2024-09-21T02:13:47.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:47 smithi161 ceph-mon[34061]: pgmap v551: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 494 op/s 2024-09-21T02:13:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:49 smithi016 ceph-mon[31245]: pgmap v552: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:13:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:49 smithi123 ceph-mon[33453]: pgmap v552: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:13:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:49 smithi161 ceph-mon[34061]: pgmap v552: 129 pgs: 129 active+clean; 16 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:13:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:51 smithi123 ceph-mon[33453]: pgmap v553: 129 pgs: 129 active+clean; 16 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 42 MiB/s wr, 687 op/s 2024-09-21T02:13:51.816 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:51 smithi016 ceph-mon[31245]: pgmap v553: 129 pgs: 129 active+clean; 16 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 42 MiB/s wr, 687 op/s 2024-09-21T02:13:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:51 smithi161 ceph-mon[34061]: pgmap v553: 129 pgs: 129 active+clean; 16 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 42 MiB/s wr, 687 op/s 2024-09-21T02:13:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:53 smithi123 ceph-mon[33453]: pgmap v554: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:13:53.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:53 smithi016 ceph-mon[31245]: pgmap v554: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:13:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:53 smithi161 ceph-mon[34061]: pgmap v554: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:13:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:55 smithi123 ceph-mon[33453]: pgmap v555: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 46 MiB/s wr, 744 op/s 2024-09-21T02:13:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:55 smithi016 ceph-mon[31245]: pgmap v555: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 46 MiB/s wr, 744 op/s 2024-09-21T02:13:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:55 smithi161 ceph-mon[34061]: pgmap v555: 129 pgs: 129 active+clean; 16 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 46 MiB/s wr, 744 op/s 2024-09-21T02:13:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:57 smithi123 ceph-mon[33453]: pgmap v556: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 38 MiB/s wr, 617 op/s 2024-09-21T02:13:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:57 smithi016 ceph-mon[31245]: pgmap v556: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 38 MiB/s wr, 617 op/s 2024-09-21T02:13:57.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:57 smithi161 ceph-mon[34061]: pgmap v556: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 38 MiB/s wr, 617 op/s 2024-09-21T02:13:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:13:59 smithi123 ceph-mon[33453]: pgmap v557: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 42 MiB/s wr, 686 op/s 2024-09-21T02:13:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:13:59 smithi016 ceph-mon[31245]: pgmap v557: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 42 MiB/s wr, 686 op/s 2024-09-21T02:13:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:13:59 smithi161 ceph-mon[34061]: pgmap v557: 129 pgs: 129 active+clean; 16 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 42 MiB/s wr, 686 op/s 2024-09-21T02:14:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:01 smithi123 ceph-mon[33453]: pgmap v558: 129 pgs: 129 active+clean; 16 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 MiB/s wr, 719 op/s 2024-09-21T02:14:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:01 smithi016 ceph-mon[31245]: pgmap v558: 129 pgs: 129 active+clean; 16 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 MiB/s wr, 719 op/s 2024-09-21T02:14:01.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:01 smithi161 ceph-mon[34061]: pgmap v558: 129 pgs: 129 active+clean; 16 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 MiB/s wr, 719 op/s 2024-09-21T02:14:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:03 smithi123 ceph-mon[33453]: pgmap v559: 129 pgs: 129 active+clean; 16 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:14:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:03 smithi016 ceph-mon[31245]: pgmap v559: 129 pgs: 129 active+clean; 16 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:14:03.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:03 smithi161 ceph-mon[34061]: pgmap v559: 129 pgs: 129 active+clean; 16 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 43 MiB/s wr, 697 op/s 2024-09-21T02:14:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:05 smithi123 ceph-mon[33453]: pgmap v560: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 47 MiB/s wr, 770 op/s 2024-09-21T02:14:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:05 smithi016 ceph-mon[31245]: pgmap v560: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 47 MiB/s wr, 770 op/s 2024-09-21T02:14:05.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:05 smithi161 ceph-mon[34061]: pgmap v560: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 47 MiB/s wr, 770 op/s 2024-09-21T02:14:07.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:07 smithi123 ceph-mon[33453]: pgmap v561: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 641 op/s 2024-09-21T02:14:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:07 smithi016 ceph-mon[31245]: pgmap v561: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 641 op/s 2024-09-21T02:14:07.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:07 smithi161 ceph-mon[34061]: pgmap v561: 129 pgs: 129 active+clean; 17 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 641 op/s 2024-09-21T02:14:09.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:09 smithi123 ceph-mon[33453]: pgmap v562: 129 pgs: 129 active+clean; 17 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 MiB/s wr, 686 op/s 2024-09-21T02:14:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:09 smithi016 ceph-mon[31245]: pgmap v562: 129 pgs: 129 active+clean; 17 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 MiB/s wr, 686 op/s 2024-09-21T02:14:09.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:09 smithi161 ceph-mon[34061]: pgmap v562: 129 pgs: 129 active+clean; 17 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 MiB/s wr, 686 op/s 2024-09-21T02:14:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:11 smithi016 ceph-mon[31245]: pgmap v563: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 42 MiB/s wr, 703 op/s 2024-09-21T02:14:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:11 smithi016 ceph-mon[31245]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T02:14:11.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:11 smithi161 ceph-mon[34061]: pgmap v563: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 42 MiB/s wr, 703 op/s 2024-09-21T02:14:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:11 smithi161 ceph-mon[34061]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T02:14:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:11 smithi123 ceph-mon[33453]: pgmap v563: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 42 MiB/s wr, 703 op/s 2024-09-21T02:14:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:11 smithi123 ceph-mon[33453]: osdmap e87: 12 total, 12 up, 12 in 2024-09-21T02:14:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:13 smithi016 ceph-mon[31245]: pgmap v565: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 40 MiB/s wr, 674 op/s 2024-09-21T02:14:13.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:13 smithi161 ceph-mon[34061]: pgmap v565: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 40 MiB/s wr, 674 op/s 2024-09-21T02:14:14.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:13 smithi123 ceph-mon[33453]: pgmap v565: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 40 MiB/s wr, 674 op/s 2024-09-21T02:14:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:15 smithi016 ceph-mon[31245]: pgmap v566: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 40 MiB/s wr, 651 op/s 2024-09-21T02:14:15.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:15 smithi161 ceph-mon[34061]: pgmap v566: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 40 MiB/s wr, 651 op/s 2024-09-21T02:14:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:15 smithi123 ceph-mon[33453]: pgmap v566: 129 pgs: 129 active+clean; 17 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 40 MiB/s wr, 651 op/s 2024-09-21T02:14:17.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:16 smithi016 ceph-mon[31245]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T02:14:17.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:16 smithi161 ceph-mon[34061]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T02:14:17.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:16 smithi123 ceph-mon[33453]: osdmap e88: 12 total, 12 up, 12 in 2024-09-21T02:14:18.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:17 smithi016 ceph-mon[31245]: pgmap v568: 129 pgs: 129 active+clean; 17 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:14:18.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:17 smithi161 ceph-mon[34061]: pgmap v568: 129 pgs: 129 active+clean; 17 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:14:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:17 smithi123 ceph-mon[33453]: pgmap v568: 129 pgs: 129 active+clean; 17 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 42 MiB/s wr, 676 op/s 2024-09-21T02:14:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:19 smithi161 ceph-mon[34061]: pgmap v569: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 39 MiB/s wr, 631 op/s 2024-09-21T02:14:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:19 smithi123 ceph-mon[33453]: pgmap v569: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 39 MiB/s wr, 631 op/s 2024-09-21T02:14:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:19 smithi016 ceph-mon[31245]: pgmap v569: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 39 MiB/s wr, 631 op/s 2024-09-21T02:14:22.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:21 smithi161 ceph-mon[34061]: pgmap v570: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T02:14:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:21 smithi123 ceph-mon[33453]: pgmap v570: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T02:14:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:21 smithi016 ceph-mon[31245]: pgmap v570: 129 pgs: 129 active+clean; 17 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 42 MiB/s wr, 675 op/s 2024-09-21T02:14:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:23 smithi123 ceph-mon[33453]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:14:24.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:23 smithi016 ceph-mon[31245]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:14:24.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:23 smithi161 ceph-mon[34061]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 41 MiB/s wr, 666 op/s 2024-09-21T02:14:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:25 smithi123 ceph-mon[33453]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 41 MiB/s wr, 658 op/s 2024-09-21T02:14:26.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:25 smithi016 ceph-mon[31245]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 41 MiB/s wr, 658 op/s 2024-09-21T02:14:26.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:25 smithi161 ceph-mon[34061]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 41 MiB/s wr, 658 op/s 2024-09-21T02:14:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:27 smithi123 ceph-mon[33453]: pgmap v573: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 41 MiB/s wr, 668 op/s 2024-09-21T02:14:28.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:27 smithi016 ceph-mon[31245]: pgmap v573: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 41 MiB/s wr, 668 op/s 2024-09-21T02:14:28.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:27 smithi161 ceph-mon[34061]: pgmap v573: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 41 MiB/s wr, 668 op/s 2024-09-21T02:14:29.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:28 smithi123 ceph-mon[33453]: pgmap v574: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 653 op/s 2024-09-21T02:14:29.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:28 smithi016 ceph-mon[31245]: pgmap v574: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 653 op/s 2024-09-21T02:14:29.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:28 smithi161 ceph-mon[34061]: pgmap v574: 129 pgs: 129 active+clean; 17 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 40 MiB/s wr, 653 op/s 2024-09-21T02:14:31.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:31 smithi016 ceph-mon[31245]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 42 MiB/s wr, 679 op/s 2024-09-21T02:14:31.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:31 smithi161 ceph-mon[34061]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 42 MiB/s wr, 679 op/s 2024-09-21T02:14:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:31 smithi123 ceph-mon[33453]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 42 MiB/s wr, 679 op/s 2024-09-21T02:14:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:33 smithi016 ceph-mon[31245]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 42 MiB/s wr, 672 op/s 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:33 smithi161 ceph-mon[34061]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 42 MiB/s wr, 672 op/s 2024-09-21T02:14:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:33 smithi123 ceph-mon[33453]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 42 MiB/s wr, 672 op/s 2024-09-21T02:14:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:35 smithi016 ceph-mon[31245]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 720 op/s 2024-09-21T02:14:35.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:35 smithi161 ceph-mon[34061]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 720 op/s 2024-09-21T02:14:35.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:35 smithi123 ceph-mon[33453]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 720 op/s 2024-09-21T02:14:37.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:37 smithi016 ceph-mon[31245]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 37 MiB/s wr, 601 op/s 2024-09-21T02:14:37.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:37 smithi161 ceph-mon[34061]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 37 MiB/s wr, 601 op/s 2024-09-21T02:14:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:37 smithi123 ceph-mon[33453]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 37 MiB/s wr, 601 op/s 2024-09-21T02:14:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:39 smithi016 ceph-mon[31245]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 MiB/s wr, 656 op/s 2024-09-21T02:14:39.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:39 smithi161 ceph-mon[34061]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 MiB/s wr, 656 op/s 2024-09-21T02:14:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:39 smithi123 ceph-mon[33453]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 MiB/s wr, 656 op/s 2024-09-21T02:14:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:41 smithi016 ceph-mon[31245]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 42 MiB/s wr, 685 op/s 2024-09-21T02:14:41.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:41 smithi161 ceph-mon[34061]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 42 MiB/s wr, 685 op/s 2024-09-21T02:14:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:41 smithi123 ceph-mon[33453]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 42 MiB/s wr, 685 op/s 2024-09-21T02:14:43.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:43 smithi016 ceph-mon[31245]: pgmap v581: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:43 smithi161 ceph-mon[34061]: pgmap v581: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:43 smithi123 ceph-mon[33453]: pgmap v581: 129 pgs: 129 active+clean; 18 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:44.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:44 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:14:44.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:44 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:14:44.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:44 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:14:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:45 smithi016 ceph-mon[31245]: pgmap v582: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 44 MiB/s wr, 714 op/s 2024-09-21T02:14:45.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:45 smithi161 ceph-mon[34061]: pgmap v582: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 44 MiB/s wr, 714 op/s 2024-09-21T02:14:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:45 smithi123 ceph-mon[33453]: pgmap v582: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 44 MiB/s wr, 714 op/s 2024-09-21T02:14:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:14:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:14:46.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:14:46.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:14:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:14:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:14:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:14:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:14:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:14:47.556 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:14:47.556 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T02:14:47.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:14:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:47 smithi123 ceph-mon[33453]: pgmap v583: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:14:47.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:47 smithi016 ceph-mon[31245]: pgmap v583: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:14:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:47 smithi161 ceph-mon[34061]: pgmap v583: 129 pgs: 129 active+clean; 18 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 36 MiB/s wr, 589 op/s 2024-09-21T02:14:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:14:49.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.661+0000 7f8cb8a35640 1 -- 172.21.15.16:0/4145203761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 msgr2=0x7f8cb410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:49.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.661+0000 7f8cb8a35640 1 --2- 172.21.15.16:0/4145203761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb410a3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009900 tx=0x7f8ca402a400 comp rx=0 tx=0).stop 2024-09-21T02:14:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 -- 172.21.15.16:0/4145203761 shutdown_connections 2024-09-21T02:14:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 --2- 172.21.15.16:0/4145203761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb411cc30 0x7f8cb411f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 --2- 172.21.15.16:0/4145203761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb410a9e0 0x7f8cb410adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 --2- 172.21.15.16:0/4145203761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb410a3c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 -- 172.21.15.16:0/4145203761 >> 172.21.15.16:0/4145203761 conn(0x7f8cb4070ec0 msgr2=0x7f8cb40712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.667+0000 7f8cb8a35640 1 -- 172.21.15.16:0/4145203761 shutdown_connections 2024-09-21T02:14:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.670+0000 7f8cb8a35640 1 -- 172.21.15.16:0/4145203761 wait complete. 2024-09-21T02:14:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.672+0000 7f8cb8a35640 1 Processor -- start 2024-09-21T02:14:49.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:49 smithi016 ceph-mon[31245]: pgmap v584: 129 pgs: 129 active+clean; 18 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 40 MiB/s wr, 650 op/s 2024-09-21T02:14:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.676+0000 7f8cb8a35640 1 -- start start 2024-09-21T02:14:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.676+0000 7f8cb8a35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb410a9e0 0x7f8cb411a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.677+0000 7f8cb8a35640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.677+0000 7f8cb8a35640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb411cc30 0x7f8cb4113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb8a35640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4122810 con 0x7f8cb410a9e0 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:44710/0 (socket says 172.21.15.16:44710) 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb1d9b640 1 -- 172.21.15.16:0/702666629 learned_addr learned my addr 172.21.15.16:0/702666629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4122690 con 0x7f8cb4111f40 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.678+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4122990 con 0x7f8cb411cc30 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 -- 172.21.15.16:0/702666629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb411cc30 msgr2=0x7f8cb4113c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb411cc30 0x7f8cb4113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 -- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb410a9e0 msgr2=0x7f8cb411a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb410a9e0 0x7f8cb411a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb41144a0 con 0x7f8cb4111f40 2024-09-21T02:14:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8cb1d9b640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800e9f0 tx=0x7f8ca800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.679+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800ccc0 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.680+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb4119140 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.680+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb4119750 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.680+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ca80035f0 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.680+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800ce60 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.684+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8ca8010690 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.686+0000 7f8ca37fe640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 0x7f8c9c0b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.686+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c940052c0 con 0x7f8cb4111f40 2024-09-21T02:14:49.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.686+0000 7f8cb259c640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 0x7f8c9c0b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:49.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.687+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9482+0+0 (secure 0 0 0) 0x7f8ca80e8dd0 con 0x7f8cb4111f40 2024-09-21T02:14:49.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.695+0000 7f8cb259c640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 0x7f8c9c0b8cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40096c0 tx=0x7f8ca4000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:49.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:49.695+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8ca80aef00 con 0x7f8cb4111f40 2024-09-21T02:14:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:49 smithi123 ceph-mon[33453]: pgmap v584: 129 pgs: 129 active+clean; 18 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 40 MiB/s wr, 650 op/s 2024-09-21T02:14:49.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:49 smithi161 ceph-mon[34061]: pgmap v584: 129 pgs: 129 active+clean; 18 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 40 MiB/s wr, 650 op/s 2024-09-21T02:14:50.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.173+0000 7f8cb8a35640 1 -- 172.21.15.16:0/702666629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c940055b0 con 0x7f8cb4111f40 2024-09-21T02:14:50.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.176+0000 7f8ca37fe640 1 -- 172.21.15.16:0/702666629 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f8ca80b32e0 con 0x7f8cb4111f40 2024-09-21T02:14:50.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:14:50.178 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:14:50.178 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 msgr2=0x7f8c9c0b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 0x7f8c9c0b8cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40096c0 tx=0x7f8ca4000bc0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 msgr2=0x7f8cb41136e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800e9f0 tx=0x7f8ca800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 shutdown_connections 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c9c0b67b0 0x7f8c9c0b8cc0 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40096c0 tx=0x7f8ca4000bc0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cb411cc30 0x7f8cb4113c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cb4111f40 0x7f8cb41136e0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800e9f0 tx=0x7f8ca800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 --2- 172.21.15.16:0/702666629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb410a9e0 0x7f8cb411a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.182+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 >> 172.21.15.16:0/702666629 conn(0x7f8cb4070ec0 msgr2=0x7f8cb411d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.187+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 shutdown_connections 2024-09-21T02:14:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:50.189+0000 7f8ca17fa640 1 -- 172.21.15.16:0/702666629 wait complete. 2024-09-21T02:14:50.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:50 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/702666629' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:14:50.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:50 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/702666629' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:14:50.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:50 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/702666629' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:14:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:51 smithi123 ceph-mon[33453]: pgmap v585: 129 pgs: 129 active+clean; 18 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 41 MiB/s wr, 681 op/s 2024-09-21T02:14:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:51 smithi016 ceph-mon[31245]: pgmap v585: 129 pgs: 129 active+clean; 18 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 41 MiB/s wr, 681 op/s 2024-09-21T02:14:51.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:51 smithi161 ceph-mon[34061]: pgmap v585: 129 pgs: 129 active+clean; 18 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 41 MiB/s wr, 681 op/s 2024-09-21T02:14:53.558 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T02:14:53.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T02:14:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:53 smithi123 ceph-mon[33453]: pgmap v586: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 38 MiB/s wr, 622 op/s 2024-09-21T02:14:53.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:53 smithi016 ceph-mon[31245]: pgmap v586: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 38 MiB/s wr, 622 op/s 2024-09-21T02:14:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:53 smithi161 ceph-mon[34061]: pgmap v586: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 38 MiB/s wr, 622 op/s 2024-09-21T02:14:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:14:55.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.044+0000 7f88631b7640 1 -- 172.21.15.16:0/2821930350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 msgr2=0x7f885c10a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.044+0000 7f88631b7640 1 --2- 172.21.15.16:0/2821930350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c10a3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8854009ca0 tx=0x7f885402a8f0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.052+0000 7f88631b7640 1 -- 172.21.15.16:0/2821930350 shutdown_connections 2024-09-21T02:14:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.052+0000 7f88631b7640 1 --2- 172.21.15.16:0/2821930350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c11efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.052+0000 7f88631b7640 1 --2- 172.21.15.16:0/2821930350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c10ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.052+0000 7f88631b7640 1 --2- 172.21.15.16:0/2821930350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c10a3c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.052+0000 7f88631b7640 1 -- 172.21.15.16:0/2821930350 >> 172.21.15.16:0/2821930350 conn(0x7f885c06ea10 msgr2=0x7f885c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.053+0000 7f88631b7640 1 -- 172.21.15.16:0/2821930350 shutdown_connections 2024-09-21T02:14:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.053+0000 7f88631b7640 1 -- 172.21.15.16:0/2821930350 wait complete. 2024-09-21T02:14:55.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.054+0000 7f88631b7640 1 Processor -- start 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 -- start start 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c11b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c114730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c114d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f885c122750 con 0x7f885c10a900 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1225d0 con 0x7f885c111d70 2024-09-21T02:14:55.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.059+0000 7f88631b7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1228d0 con 0x7f885c11cba0 2024-09-21T02:14:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.060+0000 7f88621b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c11b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.060+0000 7f88621b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c11b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47936/0 (socket says 172.21.15.16:47936) 2024-09-21T02:14:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.060+0000 7f88621b5640 1 -- 172.21.15.16:0/1696642983 learned_addr learned my addr 172.21.15.16:0/1696642983 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.060+0000 7f88619b4640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c114730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.060+0000 7f88629b6640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c114d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.063+0000 7f88629b6640 1 -- 172.21.15.16:0/1696642983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 msgr2=0x7f885c114730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.063+0000 7f88629b6640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c114730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.063+0000 7f88629b6640 1 -- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 msgr2=0x7f885c11b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.063+0000 7f88629b6640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c11b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.063+0000 7f88629b6640 1 -- 172.21.15.16:0/1696642983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c115520 con 0x7f885c11cba0 2024-09-21T02:14:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.064+0000 7f88629b6640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c114d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f885800b3e0 tx=0x7f885800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.065+0000 7f884f7fe640 1 -- 172.21.15.16:0/1696642983 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858015020 con 0x7f885c11cba0 2024-09-21T02:14:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.066+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c11a1f0 con 0x7f885c11cba0 2024-09-21T02:14:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.066+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c11a760 con 0x7f885c11cba0 2024-09-21T02:14:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.066+0000 7f884f7fe640 1 -- 172.21.15.16:0/1696642983 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8858011cc0 con 0x7f885c11cba0 2024-09-21T02:14:55.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.066+0000 7f884f7fe640 1 -- 172.21.15.16:0/1696642983 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858010d40 con 0x7f885c11cba0 2024-09-21T02:14:55.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.072+0000 7f884f7fe640 1 -- 172.21.15.16:0/1696642983 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8858018020 con 0x7f885c11cba0 2024-09-21T02:14:55.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.073+0000 7f884f7fe640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 0x7f88480b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.073+0000 7f884f7fe640 1 -- 172.21.15.16:0/1696642983 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9482+0+0 (secure 0 0 0) 0x7f88580e7ff0 con 0x7f885c11cba0 2024-09-21T02:14:55.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.074+0000 7f88621b5640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 0x7f88480b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.085+0000 7f886a98d740 1 Processor -- start 2024-09-21T02:14:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.085+0000 7f88621b5640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 0x7f88480b8da0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8854009a60 tx=0x7f885402adf0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 -- start start 2024-09-21T02:14:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f55070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f55af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ff1ecbbb0 con 0x555ff1b8db70 2024-09-21T02:14:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x555ff1ebafe0 con 0x555ff1b85990 2024-09-21T02:14:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.088+0000 7f886a98d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555ff1ebedf0 con 0x555ff1bae850 2024-09-21T02:14:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.089+0000 7f88619b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.089+0000 7f88619b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:44724/0 (socket says 172.21.15.16:44724) 2024-09-21T02:14:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.089+0000 7f88619b4640 1 -- 172.21.15.16:0/1743393109 learned_addr learned my addr 172.21.15.16:0/1743393109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.089+0000 7f88621b5640 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f55070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 -- 172.21.15.16:0/1743393109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 msgr2=0x555ff1f55070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f55070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 -- 172.21.15.16:0/1743393109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 msgr2=0x555ff1f55af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f55af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 -- 172.21.15.16:0/1743393109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ff1c28ba0 con 0x555ff1b85990 2024-09-21T02:14:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88619b4640 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f885000a830 tx=0x7f885000acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76793c19c8ce6f49 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.090+0000 7f88621b5640 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f55070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:14:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.094+0000 7f884cd9a640 1 -- 172.21.15.16:0/1743393109 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850015070 con 0x555ff1b85990 2024-09-21T02:14:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.094+0000 7f884cd9a640 1 -- 172.21.15.16:0/1743393109 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88500113f0 con 0x555ff1b85990 2024-09-21T02:14:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.095+0000 7f886a98d740 1 -- 172.21.15.16:0/1743393109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 msgr2=0x555ff1f555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.095+0000 7f886a98d740 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f885000a830 tx=0x7f885000acf0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.095+0000 7f886a98d740 1 -- 172.21.15.16:0/1743393109 shutdown_connections 2024-09-21T02:14:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.095+0000 7f886a98d740 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f55af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.095+0000 7f886a98d740 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b85990 0x555ff1f555b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.096+0000 7f886a98d740 1 --2- 172.21.15.16:0/1743393109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f55070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.096+0000 7f886a98d740 1 -- 172.21.15.16:0/1743393109 >> 172.21.15.16:0/1743393109 conn(0x555ff1ba2c40 msgr2=0x555ff1f52f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.100+0000 7f886a98d740 1 -- 172.21.15.16:0/1743393109 shutdown_connections 2024-09-21T02:14:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.102+0000 7f886a98d740 1 -- 172.21.15.16:0/1743393109 wait complete. 2024-09-21T02:14:55.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.103+0000 7f886a98d740 1 -- start start 2024-09-21T02:14:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.103+0000 7f886a98d740 1 Processor -- start 2024-09-21T02:14:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f606c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f6cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555ff1d1c7b0 con 0x555ff1bae850 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x555ff1e55e90 con 0x555ff1b8db70 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555ff1f638a0 con 0x555ff1b6d1d0 2024-09-21T02:14:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.109+0000 7f886a98d740 10 client.0 _ll_register_callbacks cb 0x555ff1eb8710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:14:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.112+0000 7f88619b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.112+0000 7f88619b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47960/0 (socket says 172.21.15.16:47960) 2024-09-21T02:14:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.112+0000 7f88619b4640 1 -- 172.21.15.16:0/2754131593 learned_addr learned my addr 172.21.15.16:0/2754131593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88629b6640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f6cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88619b4640 1 -- 172.21.15.16:0/2754131593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 msgr2=0x555ff1f6cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88619b4640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f6cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88619b4640 1 -- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 msgr2=0x555ff1f606c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88619b4640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f606c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.114+0000 7f88619b4640 1 -- 172.21.15.16:0/2754131593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ff1f613f0 con 0x555ff1bae850 2024-09-21T02:14:55.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.115+0000 7f88619b4640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8850006bd0 tx=0x7f8850010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac831893737b5348 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.119+0000 7f8845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:14:55.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.119+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850015070 con 0x555ff1bae850 2024-09-21T02:14:55.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.119+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555ff1ecdb00 con 0x555ff1bae850 2024-09-21T02:14:55.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.119+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9482+0+0 (secure 0 0 0) 0x7f8850004030 con 0x555ff1bae850 2024-09-21T02:14:55.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.120+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f885002c050 con 0x555ff1bae850 2024-09-21T02:14:55.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.121+0000 7f886a98d740 10 client.16044 fetch_fsmap learned FSMap version 28 2024-09-21T02:14:55.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.121+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ff1f6f810 con 0x555ff1bae850 2024-09-21T02:14:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.123+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x555ff1ecdb00 con 0x555ff1bae850 2024-09-21T02:14:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.123+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f8850021020 con 0x555ff1bae850 2024-09-21T02:14:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 10 client.16044 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:14:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 10 client.16044 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:14:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 20 client.16044 populate_metadata read hostname 'smithi016' 2024-09-21T02:14:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1b42580 0x555ff1f72ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 4 client.16044 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:14:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.124+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ff1ec95b0 con 0x555ff1b42580 2024-09-21T02:14:55.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.125+0000 7f88621b5640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1b42580 0x555ff1f72ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.126+0000 7f88621b5640 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1b42580 0x555ff1f72ed0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=332354fb5cb3faf4 server_cookie=5105ae0553b2b98 in_seq=0 out_seq=0 2024-09-21T02:14:55.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.127+0000 7f8845ffb640 10 client.16044 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:14:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.127+0000 7f8845ffb640 1 -- 172.21.15.16:0/2754131593 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x555ff1ec95b0 con 0x555ff1b42580 2024-09-21T02:14:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.127+0000 7f8845ffb640 10 client.16044 handle_command_reply: tid=0 2024-09-21T02:14:55.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.127+0000 7f886a98d740 1 client.16044 shutdown 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 msgr2=0x555ff1f60c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8850006bd0 tx=0x7f8850010a40 comp rx=0 tx=0).stop 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 shutdown_connections 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1b42580 0x555ff1f72ed0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f6cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1bae850 0x555ff1f60c00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 --2- 172.21.15.16:0/2754131593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 >> 172.21.15.16:0/2754131593 conn(0x555ff1ba2c40 msgr2=0x555ff1f58200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f8845ffb640 0 client.16044 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:14:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.128+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 shutdown_connections 2024-09-21T02:14:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.130+0000 7f886a98d740 1 -- 172.21.15.16:0/2754131593 wait complete. 2024-09-21T02:14:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.130+0000 7f886a98d740 20 client.16044 trim_cache size 0 max 16384 2024-09-21T02:14:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.170+0000 7f886a98d740 1 Processor -- start 2024-09-21T02:14:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.170+0000 7f886a98d740 1 -- start start 2024-09-21T02:14:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.171+0000 7f886a98d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f5ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.171+0000 7f886a98d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f6cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.171+0000 7f88621b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f5ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.171+0000 7f88621b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f5ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:42818/0 (socket says 172.21.15.16:42818) 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.171+0000 7f88621b5640 1 -- 172.21.15.16:0/3021612284 learned_addr learned my addr 172.21.15.16:0/3021612284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.172+0000 7f886a98d740 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1bae850 0x555ff1f6d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.172+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8840010150 con 0x555ff1b8db70 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.172+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8840011c90 con 0x555ff1bae850 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.172+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8840010820 con 0x555ff1b6d1d0 2024-09-21T02:14:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.172+0000 7f88619b4640 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f6cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.173+0000 7f88619b4640 1 -- 172.21.15.16:0/3021612284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 msgr2=0x555ff1f5ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.173+0000 7f88619b4640 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f5ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.173+0000 7f88619b4640 1 -- 172.21.15.16:0/3021612284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1bae850 msgr2=0x555ff1f6d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.173+0000 7f88619b4640 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1bae850 0x555ff1f6d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.173+0000 7f88619b4640 1 -- 172.21.15.16:0/3021612284 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ff1f5bf50 con 0x555ff1b8db70 2024-09-21T02:14:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.174+0000 7f88619b4640 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f6cb70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f885000c520 tx=0x7f8850010790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9347b54744805fc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.174+0000 7f8847fff640 1 -- 172.21.15.16:0/3021612284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850015040 con 0x555ff1b8db70 2024-09-21T02:14:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.174+0000 7f8847fff640 1 -- 172.21.15.16:0/3021612284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f885001add0 con 0x555ff1b8db70 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 msgr2=0x555ff1f6cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f6cb70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f885000c520 tx=0x7f8850010790 comp rx=0 tx=0).stop 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 shutdown_connections 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1bae850 0x555ff1f6d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b8db70 0x555ff1f6cb70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 --2- 172.21.15.16:0/3021612284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1b6d1d0 0x555ff1f5ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.176+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 >> 172.21.15.16:0/3021612284 conn(0x555ff1ba2c40 msgr2=0x555ff1f5fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.180+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 shutdown_connections 2024-09-21T02:14:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.180+0000 7f886a98d740 1 -- 172.21.15.16:0/3021612284 wait complete. 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.181+0000 7f886a98d740 1 -- start start 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.181+0000 7f886a98d740 1 Processor -- start 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f649e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f64fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88400115c0 con 0x555ff1b6d1d0 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8840010ef0 con 0x555ff1b8db70 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f884000e150 con 0x555ff1bae850 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.182+0000 7f886a98d740 10 client.0 _ll_register_callbacks cb 0x7f885000a6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:14:55.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.183+0000 7f88621b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.183+0000 7f88621b5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47976/0 (socket says 172.21.15.16:47976) 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.183+0000 7f88621b5640 1 -- 172.21.15.16:0/238484477 learned_addr learned my addr 172.21.15.16:0/238484477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 -- 172.21.15.16:0/238484477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 msgr2=0x555ff1f64fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f64fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 -- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 msgr2=0x555ff1f649e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f649e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 -- 172.21.15.16:0/238484477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ff1f65730 con 0x555ff1b6d1d0 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f88621b5640 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8854006010 tx=0x7f885402a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3fef778c08c211 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:14:55.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.184+0000 7f8845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:14:55.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.185+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854003d80 con 0x555ff1b6d1d0 2024-09-21T02:14:55.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.185+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f885001add0 con 0x555ff1b6d1d0 2024-09-21T02:14:55.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.185+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9482+0+0 (secure 0 0 0) 0x7f8854004040 con 0x555ff1b6d1d0 2024-09-21T02:14:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.186+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8854004240 con 0x555ff1b6d1d0 2024-09-21T02:14:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.186+0000 7f886a98d740 10 client.16050 fetch_fsmap learned FSMap version 28 2024-09-21T02:14:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.186+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ff1fdd9a0 con 0x555ff1b6d1d0 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f885001add0 con 0x555ff1b6d1d0 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f8854043040 con 0x555ff1b6d1d0 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 10 client.16050 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 10 client.16050 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 20 client.16050 populate_metadata read hostname 'smithi016' 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1fde170 0x555ff1fe0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 4 client.16050 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T02:14:55.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.196+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x555ff1ec69e0 con 0x555ff1fde170 2024-09-21T02:14:55.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.204+0000 7f88619b4640 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1fde170 0x555ff1fe0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:14:55.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.206+0000 7f88619b4640 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1fde170 0x555ff1fe0590 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e24a35ece30f561d server_cookie=61d4b0f59e0a740f in_seq=0 out_seq=0 2024-09-21T02:14:55.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.206+0000 7f8845ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f8845ffb640 1 -- 172.21.15.16:0/238484477 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555ff1ec69e0 con 0x555ff1fde170 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f8845ffb640 10 client.16050 handle_command_reply: tid=0 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 client.16050 shutdown 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 msgr2=0x555ff1f69440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8854006010 tx=0x7f885402a8f0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 shutdown_connections 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x555ff1fde170 0x555ff1fe0590 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555ff1bae850 0x555ff1f64fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x555ff1b8db70 0x555ff1f649e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f8845ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 --2- 172.21.15.16:0/238484477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555ff1b6d1d0 0x555ff1f69440 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.269+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 >> 172.21.15.16:0/238484477 conn(0x555ff1ba2c40 msgr2=0x555ff1f5fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.270+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 shutdown_connections 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.270+0000 7f886a98d740 1 -- 172.21.15.16:0/238484477 wait complete. 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.270+0000 7f886a98d740 20 client.16050 trim_cache size 0 max 16384 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 30, 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:14:55.272 INFO:teuthology.orchestra.run.smithi016.stdout: "6adf1406": { 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 30, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.1, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:14:55.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 msgr2=0x7f88480b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 0x7f88480b8da0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8854009a60 tx=0x7f885402adf0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 msgr2=0x7f885c114d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c114d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f885800b3e0 tx=0x7f885800b8a0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 shutdown_connections 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f88480b6890 0x7f88480b8da0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885c11cba0 0x7f885c114d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f885c111d70 0x7f885c114730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 --2- 172.21.15.16:0/1696642983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c10a900 0x7f885c11b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 >> 172.21.15.16:0/1696642983 conn(0x7f885c06ea10 msgr2=0x7f885c071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 shutdown_connections 2024-09-21T02:14:55.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:14:55.271+0000 7f88631b7640 1 -- 172.21.15.16:0/1696642983 wait complete. 2024-09-21T02:14:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:55 smithi123 ceph-mon[33453]: pgmap v587: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:55 smithi016 ceph-mon[31245]: pgmap v587: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:55 smithi161 ceph-mon[34061]: pgmap v587: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 41 MiB/s wr, 672 op/s 2024-09-21T02:14:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:57 smithi123 ceph-mon[33453]: pgmap v588: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 33 MiB/s wr, 547 op/s 2024-09-21T02:14:57.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:57 smithi016 ceph-mon[31245]: pgmap v588: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 33 MiB/s wr, 547 op/s 2024-09-21T02:14:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:57 smithi161 ceph-mon[34061]: pgmap v588: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 33 MiB/s wr, 547 op/s 2024-09-21T02:14:58.364 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6adf1406', quiesce took 10.8 seconds. Will release after: 10.25 2024-09-21T02:14:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:14:59 smithi123 ceph-mon[33453]: pgmap v589: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 34 MiB/s wr, 566 op/s 2024-09-21T02:14:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:14:59 smithi016 ceph-mon[31245]: pgmap v589: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 34 MiB/s wr, 566 op/s 2024-09-21T02:14:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:14:59 smithi161 ceph-mon[34061]: pgmap v589: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 34 MiB/s wr, 566 op/s 2024-09-21T02:15:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:01 smithi123 ceph-mon[33453]: pgmap v590: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 30 MiB/s wr, 496 op/s 2024-09-21T02:15:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:01 smithi016 ceph-mon[31245]: pgmap v590: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 30 MiB/s wr, 496 op/s 2024-09-21T02:15:01.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:01 smithi161 ceph-mon[34061]: pgmap v590: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 30 MiB/s wr, 496 op/s 2024-09-21T02:15:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:03 smithi123 ceph-mon[33453]: pgmap v591: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 22 MiB/s wr, 370 op/s 2024-09-21T02:15:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:03 smithi016 ceph-mon[31245]: pgmap v591: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 22 MiB/s wr, 370 op/s 2024-09-21T02:15:03.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:03 smithi161 ceph-mon[34061]: pgmap v591: 129 pgs: 129 active+clean; 18 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 22 MiB/s wr, 370 op/s 2024-09-21T02:15:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:05 smithi123 ceph-mon[33453]: pgmap v592: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-09-21T02:15:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:05 smithi016 ceph-mon[31245]: pgmap v592: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-09-21T02:15:05.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:05 smithi161 ceph-mon[34061]: pgmap v592: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 18 MiB/s wr, 324 op/s 2024-09-21T02:15:06.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:06 smithi016 ceph-mon[31245]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T02:15:06.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:06 smithi161 ceph-mon[34061]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T02:15:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:06 smithi123 ceph-mon[33453]: osdmap e89: 12 total, 12 up, 12 in 2024-09-21T02:15:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:07 smithi016 ceph-mon[31245]: pgmap v594: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 7.5 MiB/s wr, 154 op/s 2024-09-21T02:15:07.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:07 smithi161 ceph-mon[34061]: pgmap v594: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 7.5 MiB/s wr, 154 op/s 2024-09-21T02:15:08.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:07 smithi123 ceph-mon[33453]: pgmap v594: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 7.5 MiB/s wr, 154 op/s 2024-09-21T02:15:08.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:15:08.889 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.381+0000 7f9c6af07640 1 -- 172.21.15.16:0/496097844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641094a0 msgr2=0x7f9c641098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.381+0000 7f9c6af07640 1 --2- 172.21.15.16:0/496097844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641094a0 0x7f9c641098b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009900 tx=0x7f9c5002a410 comp rx=0 tx=0).stop 2024-09-21T02:15:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 -- 172.21.15.16:0/496097844 shutdown_connections 2024-09-21T02:15:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 --2- 172.21.15.16:0/496097844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 0x7f9c64110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 --2- 172.21.15.16:0/496097844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641094a0 0x7f9c641098b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 --2- 172.21.15.16:0/496097844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c64103b60 0x7f9c64103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 -- 172.21.15.16:0/496097844 >> 172.21.15.16:0/496097844 conn(0x7f9c6407b850 msgr2=0x7f9c6407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 -- 172.21.15.16:0/496097844 shutdown_connections 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 -- 172.21.15.16:0/496097844 wait complete. 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 Processor -- start 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.382+0000 7f9c6af07640 1 -- start start 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 0x7f9c6419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 0x7f9c6419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9c641139a0 con 0x7f9c64103b60 2024-09-21T02:15:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9c64113820 con 0x7f9c64109df0 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c6af07640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9c64113b20 con 0x7f9c641094a0 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52530/0 (socket says 172.21.15.16:52530) 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 -- 172.21.15.16:0/2540572445 learned_addr learned my addr 172.21.15.16:0/2540572445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c694a4640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 0x7f9c6419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c63fff640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 0x7f9c6419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 -- 172.21.15.16:0/2540572445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 msgr2=0x7f9c6419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 0x7f9c6419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 -- 172.21.15.16:0/2540572445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 msgr2=0x7f9c6419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 0x7f9c6419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c68ca3640 1 -- 172.21.15.16:0/2540572445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c641a0460 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.383+0000 7f9c63fff640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 0x7f9c6419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c68ca3640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b450 tx=0x7f9c5800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58015020 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c58011d20 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c641a0750 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58010e30 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.384+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c641a6eb0 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.385+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c64103f70 con 0x7f9c64103b60 2024-09-21T02:15:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.387+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9c58018020 con 0x7f9c64103b60 2024-09-21T02:15:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.387+0000 7f9c61ffb640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 0x7f9c540b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.388+0000 7f9c63fff640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 0x7f9c540b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.388+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9570+0+0 (secure 0 0 0) 0x7f9c580e7550 con 0x7f9c64103b60 2024-09-21T02:15:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.388+0000 7f9c63fff640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 0x7f9c540b8c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c500057c0 tx=0x7f9c50032d00 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.392+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9c5801f020 con 0x7f9c64103b60 2024-09-21T02:15:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.620+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c6402e640 con 0x7f9c64103b60 2024-09-21T02:15:09.621 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:09 smithi016 ceph-mon[31245]: pgmap v595: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 3.2 MiB/s wr, 82 op/s 2024-09-21T02:15:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.621+0000 7f9c61ffb640 1 -- 172.21.15.16:0/2540572445 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f9c5800d040 con 0x7f9c64103b60 2024-09-21T02:15:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:15:09.622 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:15:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:15:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 msgr2=0x7f9c540b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 0x7f9c540b8c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c500057c0 tx=0x7f9c50032d00 comp rx=0 tx=0).stop 2024-09-21T02:15:09.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 msgr2=0x7f9c6419a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b450 tx=0x7f9c5800b910 comp rx=0 tx=0).stop 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 shutdown_connections 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f9c540b6700 0x7f9c540b8c10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c64109df0 0x7f9c6419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c641094a0 0x7f9c6419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 --2- 172.21.15.16:0/2540572445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64103b60 0x7f9c6419a4f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 >> 172.21.15.16:0/2540572445 conn(0x7f9c6407b850 msgr2=0x7f9c64107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.623+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 shutdown_connections 2024-09-21T02:15:09.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:09.624+0000 7f9c6af07640 1 -- 172.21.15.16:0/2540572445 wait complete. 2024-09-21T02:15:09.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:09 smithi161 ceph-mon[34061]: pgmap v595: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 3.2 MiB/s wr, 82 op/s 2024-09-21T02:15:10.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:09 smithi123 ceph-mon[33453]: pgmap v595: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 3.2 MiB/s wr, 82 op/s 2024-09-21T02:15:10.444 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 6adf1406 --release --await' 2024-09-21T02:15:10.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 6adf1406 --release --await 2024-09-21T02:15:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:10.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:10 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2540572445' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:10.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:10 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2540572445' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:11.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:10 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2540572445' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 -- 172.21.15.16:0/1756951975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23501094a0 msgr2=0x7f23501098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 --2- 172.21.15.16:0/1756951975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23501094a0 0x7f23501098b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2344009900 tx=0x7f234402a410 comp rx=0 tx=0).stop 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 -- 172.21.15.16:0/1756951975 shutdown_connections 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 --2- 172.21.15.16:0/1756951975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350109df0 0x7f2350110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 --2- 172.21.15.16:0/1756951975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23501094a0 0x7f23501098b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 --2- 172.21.15.16:0/1756951975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2350103b60 0x7f2350103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 -- 172.21.15.16:0/1756951975 >> 172.21.15.16:0/1756951975 conn(0x7f235007b850 msgr2=0x7f235007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 -- 172.21.15.16:0/1756951975 shutdown_connections 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.219+0000 7f235611d640 1 -- 172.21.15.16:0/1756951975 wait complete. 2024-09-21T02:15:11.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 Processor -- start 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 -- start start 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350103b60 0x7f235019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 0x7f235019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2350113960 con 0x7f2350109df0 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23501137e0 con 0x7f2350103b60 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f235611d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2350113ae0 con 0x7f23501094a0 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f234ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f234ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52542/0 (socket says 172.21.15.16:52542) 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f234ffff640 1 -- 172.21.15.16:0/2699889327 learned_addr learned my addr 172.21.15.16:0/2699889327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f234f7fe640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350103b60 0x7f235019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.220+0000 7f234effd640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 0x7f235019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 -- 172.21.15.16:0/2699889327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 msgr2=0x7f235019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 0x7f235019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 -- 172.21.15.16:0/2699889327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350103b60 msgr2=0x7f235019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350103b60 0x7f235019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 -- 172.21.15.16:0/2699889327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23501a0340 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234effd640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 0x7f235019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234ffff640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f234800eaa0 tx=0x7f234800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234cff9640 1 -- 172.21.15.16:0/2699889327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234800cdb0 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23501a0630 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234cff9640 1 -- 172.21.15.16:0/2699889327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2348005020 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f234cff9640 1 -- 172.21.15.16:0/2699889327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23480116d0 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.221+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23501a6d90 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 Processor -- start 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f234cff9640 1 -- 172.21.15.16:0/2699889327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f23480051d0 con 0x7f2350109df0 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 -- start start 2024-09-21T02:15:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc600e3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cc5ffcedc0 con 0x55cc5fd1c840 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cc6005b1f0 con 0x55cc5fd3d520 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f235c691740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cc6004a620 con 0x55cc5fd14660 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52548/0 (socket says 172.21.15.16:52548) 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.224+0000 7f234effd640 1 -- 172.21.15.16:0/3961852276 learned_addr learned my addr 172.21.15.16:0/3961852276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc600e3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234f7fe640 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 -- 172.21.15.16:0/3961852276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 msgr2=0x55cc600e3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600e3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234cff9640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 0x7f23380b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 -- 172.21.15.16:0/3961852276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 msgr2=0x55cc600e3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc600e3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 -- 172.21.15.16:0/3961852276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc60012fd0 con 0x55cc5fd1c840 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234ffff640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 0x7f23380b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234effd640 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23440096c0 tx=0x7f234402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71d3c40a44c1aef server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f234cff9640 1 -- 172.21.15.16:0/2699889327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9494+0+0 (secure 0 0 0) 0x7f23480e7cd0 con 0x7f2350109df0 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f2335ffb640 1 -- 172.21.15.16:0/3961852276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344035040 con 0x55cc5fd1c840 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f2335ffb640 1 -- 172.21.15.16:0/3961852276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23440044c0 con 0x55cc5fd1c840 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.225+0000 7f2335ffb640 1 -- 172.21.15.16:0/3961852276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344003b60 con 0x55cc5fd1c840 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 -- 172.21.15.16:0/3961852276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 msgr2=0x55cc600e3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f234ffff640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 0x7f23380b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f23480a9d20 tx=0x7f234807a400 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23440096c0 tx=0x7f234402a410 comp rx=0 tx=0).stop 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 -- 172.21.15.16:0/3961852276 shutdown_connections 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc600e3f90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600e3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 --2- 172.21.15.16:0/3961852276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc600e3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 -- 172.21.15.16:0/3961852276 >> 172.21.15.16:0/3961852276 conn(0x55cc5fd31910 msgr2=0x55cc600e1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 -- 172.21.15.16:0/3961852276 shutdown_connections 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.226+0000 7f235c691740 1 -- 172.21.15.16:0/3961852276 wait complete. 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 -- start start 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 Processor -- start 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd14660 0x55cc600f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd1c840 0x55cc600f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cc5ffe5300 con 0x55cc5fd3d520 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cc6004e430 con 0x55cc5fd14660 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cc5feabbb0 con 0x55cc5fd1c840 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f235c691740 10 client.0 _ll_register_callbacks cb 0x55cc5ffff120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.227+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52550/0 (socket says 172.21.15.16:52550) 2024-09-21T02:15:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 -- 172.21.15.16:0/969080200 learned_addr learned my addr 172.21.15.16:0/969080200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234f7fe640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd1c840 0x55cc600f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd14660 0x55cc600f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 -- 172.21.15.16:0/969080200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd1c840 msgr2=0x55cc600f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd1c840 0x55cc600f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 -- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd14660 msgr2=0x55cc600f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd14660 0x55cc600f2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 -- 172.21.15.16:0/969080200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc5ffd4780 con 0x55cc5fd3d520 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f234effd640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2344032fd0 tx=0x7f2344032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c6d30993789f53 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f232effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344035040 con 0x55cc5fd3d520 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc6005d140 con 0x55cc5fd3d520 2024-09-21T02:15:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9494+0+0 (secure 0 0 0) 0x7f2344003940 con 0x55cc5fd3d520 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2344000e60 con 0x55cc5fd3d520 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f235c691740 10 client.16068 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.228+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc5fda8b00 con 0x55cc5fd3d520 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55cc6005d140 con 0x55cc5fd3d520 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f234403a040 con 0x55cc5fd3d520 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 10 client.16068 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 10 client.16068 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 20 client.16068 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcfbea0 0x55cc60105070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 4 client.16068 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:15:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc60058bf0 con 0x55cc5fcfbea0 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.229+0000 7f234f7fe640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcfbea0 0x55cc60105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.230+0000 7f234f7fe640 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcfbea0 0x55cc60105070 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c7529df2e517c4 server_cookie=1f648922053cad54 in_seq=0 out_seq=0 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.230+0000 7f232effd640 10 client.16068 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.230+0000 7f232effd640 1 -- 172.21.15.16:0/969080200 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cc60058bf0 con 0x55cc5fcfbea0 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.230+0000 7f232effd640 10 client.16068 handle_command_reply: tid=0 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 client.16068 shutdown 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 msgr2=0x55cc600f3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2344032fd0 tx=0x7f2344032c90 comp rx=0 tx=0).stop 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 shutdown_connections 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcfbea0 0x55cc60105070 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f3a50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f232effd640 0 client.16068 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd1c840 0x55cc600f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 --2- 172.21.15.16:0/969080200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd14660 0x55cc600f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 >> 172.21.15.16:0/969080200 conn(0x55cc5fd31910 msgr2=0x55cc600e4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 shutdown_connections 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 1 -- 172.21.15.16:0/969080200 wait complete. 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.231+0000 7f235c691740 20 client.16068 trim_cache size 0 max 16384 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 Processor -- start 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 -- start start 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600dffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 0x55cc600e0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23501aec60 con 0x55cc5fd3d520 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23501b07a0 con 0x55cc5fd1c840 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.248+0000 7f235c691740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23501af330 con 0x55cc5fd14660 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52556/0 (socket says 172.21.15.16:52556) 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 -- 172.21.15.16:0/4194850202 learned_addr learned my addr 172.21.15.16:0/4194850202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600dffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234f7fe640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 0x55cc600e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 -- 172.21.15.16:0/4194850202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 msgr2=0x55cc600dffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600dffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 -- 172.21.15.16:0/4194850202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 msgr2=0x55cc600e0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 0x55cc600e0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 -- 172.21.15.16:0/4194850202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc5fe1f960 con 0x55cc5fd3d520 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234f7fe640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 0x55cc600e0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f234effd640 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2344009900 tx=0x7f234404d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3756f3df7dee992b server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f23357fa640 1 -- 172.21.15.16:0/4194850202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344035070 con 0x55cc5fd3d520 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f23357fa640 1 -- 172.21.15.16:0/4194850202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2344031db0 con 0x55cc5fd3d520 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.249+0000 7f23357fa640 1 -- 172.21.15.16:0/4194850202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234403d680 con 0x55cc5fd3d520 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 -- 172.21.15.16:0/4194850202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 msgr2=0x55cc600f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2344009900 tx=0x7f234404d040 comp rx=0 tx=0).stop 2024-09-21T02:15:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 -- 172.21.15.16:0/4194850202 shutdown_connections 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd3d520 0x55cc600f4e70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd1c840 0x55cc600e0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 --2- 172.21.15.16:0/4194850202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc600dffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 -- 172.21.15.16:0/4194850202 >> 172.21.15.16:0/4194850202 conn(0x55cc5fd31910 msgr2=0x55cc600edc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 -- 172.21.15.16:0/4194850202 shutdown_connections 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.250+0000 7f235c691740 1 -- 172.21.15.16:0/4194850202 wait complete. 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 -- start start 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 Processor -- start 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc60105fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc60106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23501b00d0 con 0x55cc5fd1c840 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23501afa00 con 0x55cc5fd3d520 2024-09-21T02:15:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23501acc40 con 0x55cc5fd14660 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f235c691740 10 client.0 _ll_register_callbacks cb 0x7f2344032d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f234f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f234f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52558/0 (socket says 172.21.15.16:52558) 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f234f7fe640 1 -- 172.21.15.16:0/231993491 learned_addr learned my addr 172.21.15.16:0/231993491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.251+0000 7f234effd640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc60106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234effd640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc60105fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 -- 172.21.15.16:0/231993491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 msgr2=0x55cc60105fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc60105fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 -- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 msgr2=0x55cc60106a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc60106a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 -- 172.21.15.16:0/231993491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6016e7f0 con 0x55cc5fd1c840 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234effd640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc60106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f234f7fe640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2340000c00 tx=0x7f234000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6f804d1ce32d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f232effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340029060 con 0x55cc5fd1c840 2024-09-21T02:15:11.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9494+0+0 (secure 0 0 0) 0x7f2340024460 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2344031db0 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2340030050 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f235c691740 10 client.16074 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.252+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc6016f9d0 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2344031db0 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f234001d480 con 0x55cc5fd1c840 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 10 client.16074 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 10 client.16074 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 20 client.16074 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcd1250 0x55cc60172170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 4 client.16074 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6adf1406", "release": true, "await": true}] 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6adf1406", "release": true, "await": true}) -- 0x55cc60056020 con 0x55cc5fcd1250 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f234effd640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcd1250 0x55cc60172170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:11.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f234effd640 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcd1250 0x55cc60172170 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ff183aef2335210 server_cookie=e819359e7dbd5b87 in_seq=0 out_seq=0 2024-09-21T02:15:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.253+0000 7f232effd640 10 client.16074 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.259+0000 7f232effd640 1 -- 172.21.15.16:0/231993491 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cc60056020 con 0x55cc5fcd1250 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.259+0000 7f232effd640 10 client.16074 handle_command_reply: tid=0 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.259+0000 7f235c691740 1 client.16074 shutdown 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 msgr2=0x55cc60106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2340000c00 tx=0x7f234000bd90 comp rx=0 tx=0).stop 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 shutdown_connections 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cc5fcd1250 0x55cc60172170 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cc5fd3d520 0x55cc60106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f232effd640 0 client.16074 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cc5fd1c840 0x55cc60106500 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 --2- 172.21.15.16:0/231993491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cc5fd14660 0x55cc60105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 >> 172.21.15.16:0/231993491 conn(0x55cc5fd31910 msgr2=0x55cc600fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 shutdown_connections 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 1 -- 172.21.15.16:0/231993491 wait complete. 2024-09-21T02:15:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.260+0000 7f235c691740 20 client.16074 trim_cache size 0 max 16384 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 32, 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "6adf1406": { 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 32, 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:15:11.276 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.1, 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:11.277 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 msgr2=0x7f23380b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 0x7f23380b8b90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f23480a9d20 tx=0x7f234807a400 comp rx=0 tx=0).stop 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 msgr2=0x7f235019fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f234800eaa0 tx=0x7f234800ef60 comp rx=0 tx=0).stop 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 shutdown_connections 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23380b6680 0x7f23380b8b90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2350109df0 0x7f235019fc10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23501094a0 0x7f235019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 --2- 172.21.15.16:0/2699889327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2350103b60 0x7f235019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 >> 172.21.15.16:0/2699889327 conn(0x7f235007b850 msgr2=0x7f23501055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 shutdown_connections 2024-09-21T02:15:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:11.261+0000 7f235611d640 1 -- 172.21.15.16:0/2699889327 wait complete. 2024-09-21T02:15:12.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:11 smithi016 ceph-mon[31245]: pgmap v596: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 876 KiB/s wr, 43 op/s 2024-09-21T02:15:12.131 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:11 smithi016 ceph-mon[31245]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T02:15:12.136 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6adf1406', total seconds elapsed: 24.6 2024-09-21T02:15:12.136 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.05 2024-09-21T02:15:12.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:15:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:11 smithi123 ceph-mon[33453]: pgmap v596: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 876 KiB/s wr, 43 op/s 2024-09-21T02:15:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:11 smithi123 ceph-mon[33453]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T02:15:12.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:11 smithi161 ceph-mon[34061]: pgmap v596: 129 pgs: 3 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 125 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 876 KiB/s wr, 43 op/s 2024-09-21T02:15:12.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:11 smithi161 ceph-mon[34061]: osdmap e90: 12 total, 12 up, 12 in 2024-09-21T02:15:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:13 smithi123 ceph-mon[33453]: pgmap v598: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 160 KiB/s wr, 10 op/s 2024-09-21T02:15:14.325 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:13 smithi016 ceph-mon[31245]: pgmap v598: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 160 KiB/s wr, 10 op/s 2024-09-21T02:15:14.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:13 smithi161 ceph-mon[34061]: pgmap v598: 129 pgs: 129 active+clean; 18 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 160 KiB/s wr, 10 op/s 2024-09-21T02:15:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:15 smithi123 ceph-mon[33453]: pgmap v599: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.9 MiB/s wr, 111 op/s 2024-09-21T02:15:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:15 smithi123 ceph-mon[33453]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T02:15:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:15 smithi016 ceph-mon[31245]: pgmap v599: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.9 MiB/s wr, 111 op/s 2024-09-21T02:15:16.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:16 smithi016 ceph-mon[31245]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T02:15:16.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:15 smithi161 ceph-mon[34061]: pgmap v599: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 5.9 MiB/s wr, 111 op/s 2024-09-21T02:15:16.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:15 smithi161 ceph-mon[34061]: osdmap e91: 12 total, 12 up, 12 in 2024-09-21T02:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.833+0000 7f211a3a8640 1 -- 172.21.15.16:0/1395938320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21141083a0 msgr2=0x7f21141087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.833+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1395938320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21141083a0 0x7f21141087b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2104009900 tx=0x7f210402a410 comp rx=0 tx=0).stop 2024-09-21T02:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 -- 172.21.15.16:0/1395938320 shutdown_connections 2024-09-21T02:15:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1395938320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f211410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1395938320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21141083a0 0x7f21141087b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1395938320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2114102a60 0x7f2114102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 -- 172.21.15.16:0/1395938320 >> 172.21.15.16:0/1395938320 conn(0x7f2114079730 msgr2=0x7f2114079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.836+0000 7f211a3a8640 1 -- 172.21.15.16:0/1395938320 shutdown_connections 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.838+0000 7f211a3a8640 1 -- 172.21.15.16:0/1395938320 wait complete. 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.839+0000 7f211a3a8640 1 Processor -- start 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.840+0000 7f211a3a8640 1 -- start start 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2114102a60 0x7f2114113510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21141083a0 0x7f2114113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2114111830 con 0x7f21141083a0 2024-09-21T02:15:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f21141116b0 con 0x7f2114108cf0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f211a3a8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21141119b0 con 0x7f2114102a60 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f2118945640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f2118945640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:45724/0 (socket says 172.21.15.16:45724) 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f2118945640 1 -- 172.21.15.16:0/1056433484 learned_addr learned my addr 172.21.15.16:0/1056433484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f2113fff640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2114102a60 0x7f2114113510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.841+0000 7f21137fe640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21141083a0 0x7f2114113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 -- 172.21.15.16:0/1056433484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2114102a60 msgr2=0x7f2114113510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2114102a60 0x7f2114113510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 -- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21141083a0 msgr2=0x7f2114113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21141083a0 0x7f2114113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 -- 172.21.15.16:0/1056433484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2114119480 con 0x7f2114108cf0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f2118945640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f210800b3f0 tx=0x7f210800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108015020 con 0x7f2114108cf0 2024-09-21T02:15:17.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.842+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2114119710 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.843+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211411dee0 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.843+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2108011cc0 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.843+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108010db0 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.844+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2100005290 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.846+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2108018020 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.846+0000 7f21117fa640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 0x7f20fc0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.847+0000 7f2113fff640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 0x7f20fc0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.847+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f21080e7510 con 0x7f2114108cf0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.847+0000 7f2113fff640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 0x7f20fc0b8c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2114119de0 tx=0x7f210c00d040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:17.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:17.850+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f210801f020 con 0x7f2114108cf0 2024-09-21T02:15:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:17 smithi123 ceph-mon[33453]: pgmap v601: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 9.3 MiB/s wr, 167 op/s 2024-09-21T02:15:18.305 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:17 smithi016 ceph-mon[31245]: pgmap v601: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 9.3 MiB/s wr, 167 op/s 2024-09-21T02:15:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.302+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2100005580 con 0x7f2114108cf0 2024-09-21T02:15:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.305+0000 7f21117fa640 1 -- 172.21.15.16:0/1056433484 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f21080ad6b0 con 0x7f2114108cf0 2024-09-21T02:15:18.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:15:18.307 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:15:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:15:18.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 msgr2=0x7f20fc0b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:18.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 0x7f20fc0b8c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2114119de0 tx=0x7f210c00d040 comp rx=0 tx=0).stop 2024-09-21T02:15:18.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 msgr2=0x7f2114118d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:18.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f210800b3f0 tx=0x7f210800b8b0 comp rx=0 tx=0).stop 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 shutdown_connections 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f20fc0b6730 0x7f20fc0b8c40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.310+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2114108cf0 0x7f2114118d50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.311+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21141083a0 0x7f2114113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.311+0000 7f211a3a8640 1 --2- 172.21.15.16:0/1056433484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2114102a60 0x7f2114113510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.311+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 >> 172.21.15.16:0/1056433484 conn(0x7f2114079730 msgr2=0x7f21141052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.315+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 shutdown_connections 2024-09-21T02:15:18.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:18.316+0000 7f211a3a8640 1 -- 172.21.15.16:0/1056433484 wait complete. 2024-09-21T02:15:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:17 smithi161 ceph-mon[34061]: pgmap v601: 129 pgs: 129 active+clean; 18 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 9.3 MiB/s wr, 167 op/s 2024-09-21T02:15:19.082 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:18 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1056433484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:19.159 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0' 2024-09-21T02:15:19.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 141.05 --await-for 45.0 2024-09-21T02:15:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:18 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1056433484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:19.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:18 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1056433484' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:19.813 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:20.208 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:19 smithi016 ceph-mon[31245]: pgmap v602: 129 pgs: 129 active+clean; 19 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 16 MiB/s wr, 268 op/s 2024-09-21T02:15:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:19 smithi123 ceph-mon[33453]: pgmap v602: 129 pgs: 129 active+clean; 19 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 16 MiB/s wr, 268 op/s 2024-09-21T02:15:20.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:19 smithi161 ceph-mon[34061]: pgmap v602: 129 pgs: 129 active+clean; 19 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 16 MiB/s wr, 268 op/s 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 -- 172.21.15.16:0/994100414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 msgr2=0x7f19e41098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 --2- 172.21.15.16:0/994100414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e41098f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009900 tx=0x7f19d002a410 comp rx=0 tx=0).stop 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 -- 172.21.15.16:0/994100414 shutdown_connections 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 --2- 172.21.15.16:0/994100414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 0x7f19e4110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 --2- 172.21.15.16:0/994100414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e41098f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 --2- 172.21.15.16:0/994100414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 -- 172.21.15.16:0/994100414 >> 172.21.15.16:0/994100414 conn(0x7f19e40ff5f0 msgr2=0x7f19e4101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 -- 172.21.15.16:0/994100414 shutdown_connections 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.532+0000 7f19eab92640 1 -- 172.21.15.16:0/994100414 wait complete. 2024-09-21T02:15:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 Processor -- start 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 -- start start 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 0x7f19e4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41139f0 con 0x7f19e41094e0 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.533+0000 7f19eab92640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4113870 con 0x7f19e4103ba0 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19eab92640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4113b70 con 0x7f19e4109e30 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:45734/0 (socket says 172.21.15.16:45734) 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 -- 172.21.15.16:0/2369695119 learned_addr learned my addr 172.21.15.16:0/2369695119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e912f640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 0x7f19e4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 -- 172.21.15.16:0/2369695119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 msgr2=0x7f19e4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 0x7f19e4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 -- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 msgr2=0x7f19e4075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e4075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 -- 172.21.15.16:0/2369695119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e41a8680 con 0x7f19e4103ba0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e892e640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19d800e9f0 tx=0x7f19d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e1ffb640 1 -- 172.21.15.16:0/2369695119 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d800cd80 con 0x7f19e4103ba0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e1ffb640 1 -- 172.21.15.16:0/2369695119 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d80108c0 con 0x7f19e4103ba0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e41a8970 con 0x7f19e4103ba0 2024-09-21T02:15:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.534+0000 7f19e1ffb640 1 -- 172.21.15.16:0/2369695119 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80187c0 con 0x7f19e4103ba0 2024-09-21T02:15:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.535+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e41a8e80 con 0x7f19e4103ba0 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.537+0000 7f19f1106740 1 Processor -- start 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.537+0000 7f19f1106740 1 -- start start 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.537+0000 7f19f1106740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.537+0000 7f19f1106740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19f1106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 0x558cfb1e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19f1106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cfb15dbb0 con 0x558cfae1fb70 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19f1106740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558cfb14cfe0 con 0x558cfae40850 2024-09-21T02:15:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19f1106740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558cfb150df0 con 0x558cfae17990 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e1ffb640 1 -- 172.21.15.16:0/2369695119 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f19d8010430 con 0x7f19e4103ba0 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 0x558cfb1e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:45748/0 (socket says 172.21.15.16:45748) 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 0x558cfb1e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46236/0 (socket says 172.21.15.16:46236) 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 -- 172.21.15.16:0/1644319424 learned_addr learned my addr 172.21.15.16:0/1644319424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e892e640 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 -- 172.21.15.16:0/1644319424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 msgr2=0x558cfb1e73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1e73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 -- 172.21.15.16:0/1644319424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 msgr2=0x558cfb1e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 0x558cfb1e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 -- 172.21.15.16:0/1644319424 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cfaebaba0 con 0x558cfae40850 2024-09-21T02:15:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.538+0000 7f19e912f640 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009fd0 tx=0x7f19dc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e48d35905340371 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19ceffd640 1 -- 172.21.15.16:0/1644319424 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc004d20 con 0x558cfae40850 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19ceffd640 1 -- 172.21.15.16:0/1644319424 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19dc004050 con 0x558cfae40850 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19ceffd640 1 -- 172.21.15.16:0/1644319424 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc005260 con 0x558cfae40850 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19e1ffb640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 0x7f19d40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19e892e640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 0x7f19d40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.539+0000 7f19e1ffb640 1 -- 172.21.15.16:0/2369695119 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f19d80e7e80 con 0x7f19e4103ba0 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- 172.21.15.16:0/1644319424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 msgr2=0x558cfb1e6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19e892e640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 0x7f19d40b8b90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x558cfb1e5af0 tx=0x7f19d8000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009fd0 tx=0x7f19dc0118d0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- 172.21.15.16:0/1644319424 shutdown_connections 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae1fb70 0x558cfb1e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 --2- 172.21.15.16:0/1644319424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae40850 0x558cfb1e6e70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- 172.21.15.16:0/1644319424 >> 172.21.15.16:0/1644319424 conn(0x558cfae34c40 msgr2=0x558cfb1e4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- 172.21.15.16:0/1644319424 shutdown_connections 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- 172.21.15.16:0/1644319424 wait complete. 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 -- start start 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.540+0000 7f19f1106740 1 Processor -- start 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae40850 0x558cfb1fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558cfafae7b0 con 0x558cfae40850 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x558cfb0e7e90 con 0x558cfae1fb70 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x558cfb1f5650 con 0x558cfae17990 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19f1106740 10 client.0 _ll_register_callbacks cb 0x558cfb14a710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:45754/0 (socket says 172.21.15.16:45754) 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 -- 172.21.15.16:0/3547344086 learned_addr learned my addr 172.21.15.16:0/3547344086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e912f640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae40850 0x558cfb1fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e912f640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 -- 172.21.15.16:0/3547344086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 msgr2=0x558cfb1f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 -- 172.21.15.16:0/3547344086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae40850 msgr2=0x558cfb1fe940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae40850 0x558cfb1fe940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.541+0000 7f19e3fff640 1 -- 172.21.15.16:0/3547344086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cfb1fef50 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005ab0 tx=0x7f19d0004440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=944051fcc48e8dbf server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0035040 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f19d0003a00 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19dc004050 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19d004e050 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 10 client.25682 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558cfb201880 con 0x558cfae1fb70 2024-09-21T02:15:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19dc004050 con 0x558cfae1fb70 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f19d0031400 con 0x558cfae1fb70 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 10 client.25682 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 10 client.25682 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 20 client.25682 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.542+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfadff1d0 0x558cfb204df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.543+0000 7f19f1106740 4 client.25682 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.543+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cfb15fb00 con 0x558cfadff1d0 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.543+0000 7f19e912f640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfadff1d0 0x558cfb204df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.543+0000 7f19c3fff640 10 client.25682 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.543+0000 7f19e912f640 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfadff1d0 0x558cfb204df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f88daa68574c667 server_cookie=f5049daeb009ef0d in_seq=0 out_seq=0 2024-09-21T02:15:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19c3fff640 1 -- 172.21.15.16:0/3547344086 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558cfb15fb00 con 0x558cfadff1d0 2024-09-21T02:15:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19c3fff640 10 client.25682 handle_command_reply: tid=0 2024-09-21T02:15:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 client.25682 shutdown 2024-09-21T02:15:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 msgr2=0x558cfb1f3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005ab0 tx=0x7f19d0004440 comp rx=0 tx=0).stop 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 shutdown_connections 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfadff1d0 0x558cfb204df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19c3fff640 0 client.25682 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae40850 0x558cfb1fe940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f3000 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 --2- 172.21.15.16:0/3547344086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfae17990 0x558cfb1f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 >> 172.21.15.16:0/3547344086 conn(0x558cfae34c40 msgr2=0x558cfb1ea050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 shutdown_connections 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 1 -- 172.21.15.16:0/3547344086 wait complete. 2024-09-21T02:15:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.544+0000 7f19f1106740 20 client.25682 trim_cache size 0 max 16384 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.567+0000 7f19f1106740 1 Processor -- start 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.567+0000 7f19f1106740 1 -- start start 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.567+0000 7f19f1106740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.567+0000 7f19f1106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.567+0000 7f19e912f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46258/0 (socket says 172.21.15.16:46258) 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:40720/0 (socket says 172.21.15.16:40720) 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e3fff640 1 -- 172.21.15.16:0/1513108013 learned_addr learned my addr 172.21.15.16:0/1513108013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19f1106740 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b3030 con 0x558cfae17990 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b4b70 con 0x558cfae1fb70 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b3700 con 0x558cfadff1d0 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 -- 172.21.15.16:0/1513108013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 msgr2=0x558cfb1ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 -- 172.21.15.16:0/1513108013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 msgr2=0x558cfb1f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.568+0000 7f19e912f640 1 -- 172.21.15.16:0/1513108013 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cfb1e52a0 con 0x558cfadff1d0 2024-09-21T02:15:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.575+0000 7f19e912f640 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009fa0 tx=0x7f19dc009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4893864878600db server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.579+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1513108013 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0046f0 con 0x558cfadff1d0 2024-09-21T02:15:20.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.579+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1513108013 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19dc003ec0 con 0x558cfadff1d0 2024-09-21T02:15:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.584+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1513108013 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00fcf0 con 0x558cfadff1d0 2024-09-21T02:15:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.585+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 msgr2=0x558cfb1f2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.585+0000 7f19f1106740 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009fa0 tx=0x7f19dc009500 comp rx=0 tx=0).stop 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 shutdown_connections 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 --2- 172.21.15.16:0/1513108013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1f2110 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 >> 172.21.15.16:0/1513108013 conn(0x558cfae34c40 msgr2=0x558cfb1f3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 shutdown_connections 2024-09-21T02:15:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.588+0000 7f19f1106740 1 -- 172.21.15.16:0/1513108013 wait complete. 2024-09-21T02:15:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.589+0000 7f19f1106740 1 -- start start 2024-09-21T02:15:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.589+0000 7f19f1106740 1 Processor -- start 2024-09-21T02:15:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b44a0 con 0x558cfae17990 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b3dd0 con 0x558cfae1fb70 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b1030 con 0x558cfadff1d0 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.591+0000 7f19f1106740 10 client.0 _ll_register_callbacks cb 0x7f19d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.593+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.593+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46272/0 (socket says 172.21.15.16:46272) 2024-09-21T02:15:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.593+0000 7f19e3fff640 1 -- 172.21.15.16:0/3032108529 learned_addr learned my addr 172.21.15.16:0/3032108529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.593+0000 7f19e912f640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.593+0000 7f19e912f640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.595+0000 7f19e3fff640 1 -- 172.21.15.16:0/3032108529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 msgr2=0x558cfb1faf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.595+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1faf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.595+0000 7f19e3fff640 1 -- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 msgr2=0x558cfb1f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.595+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.595+0000 7f19e3fff640 1 -- 172.21.15.16:0/3032108529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cfb1f63e0 con 0x558cfae17990 2024-09-21T02:15:20.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.602+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f19d0031850 tx=0x7f19d0031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28d7fb80544d787 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:20.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.603+0000 7f19c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:15:20.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.607+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0057060 con 0x558cfae17990 2024-09-21T02:15:20.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.607+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f19d0003800 con 0x558cfae17990 2024-09-21T02:15:20.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.607+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19dc003ec0 con 0x558cfae17990 2024-09-21T02:15:20.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.611+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19d0035040 con 0x558cfae17990 2024-09-21T02:15:20.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.611+0000 7f19f1106740 10 client.16095 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:20.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.611+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558cfb2925b0 con 0x558cfae17990 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.615+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19dc003ec0 con 0x558cfae17990 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.615+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f19d003c6c0 con 0x558cfae17990 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 10 client.16095 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 10 client.16095 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 20 client.16095 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:20.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfae40850 0x558cfb294de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:20.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 4 client.16095 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}] 2024-09-21T02:15:20.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.616+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.05, "await_for": 45.0}) -- 0x558cfb15b5b0 con 0x558cfae40850 2024-09-21T02:15:20.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.618+0000 7f19e912f640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfae40850 0x558cfb294de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:20.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.624+0000 7f19e912f640 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfae40850 0x558cfb294de0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a60c887a699f936 server_cookie=f00981b8c024f2d5 in_seq=0 out_seq=0 2024-09-21T02:15:20.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.625+0000 7f19c3fff640 10 client.16095 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19c3fff640 1 -- 172.21.15.16:0/3032108529 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f19dc03f260 con 0x558cfae40850 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19c3fff640 10 client.16095 handle_command_reply: tid=0 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 client.16095 shutdown 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 msgr2=0x558cfb1f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f19d0031850 tx=0x7f19d0031630 comp rx=0 tx=0).stop 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 shutdown_connections 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x558cfae40850 0x558cfb294de0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558cfae1fb70 0x558cfb1f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558cfae17990 0x558cfb1f5860 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19c3fff640 0 client.16095 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 --2- 172.21.15.16:0/3032108529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558cfadff1d0 0x558cfb1faf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 >> 172.21.15.16:0/3032108529 conn(0x558cfae34c40 msgr2=0x558cfb1f3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 shutdown_connections 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 1 -- 172.21.15.16:0/3032108529 wait complete. 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.682+0000 7f19f1106740 20 client.16095 trim_cache size 0 max 16384 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:15:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 34, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "8fe5d51f": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 34, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.1, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:20.687 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 msgr2=0x7f19d40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 0x7f19d40b8b90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x558cfb1e5af0 tx=0x7f19d8000bc0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 msgr2=0x7f19e4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19d800e9f0 tx=0x7f19d800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 shutdown_connections 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b6680 0x7f19d40b8b90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4109e30 0x7f19e4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094e0 0x7f19e4075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 --2- 172.21.15.16:0/2369695119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4103ba0 0x7f19e4075710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 >> 172.21.15.16:0/2369695119 conn(0x7f19e40ff5f0 msgr2=0x7f19e4101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 shutdown_connections 2024-09-21T02:15:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:20.683+0000 7f19eab92640 1 -- 172.21.15.16:0/2369695119 wait complete. 2024-09-21T02:15:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:21 smithi123 ceph-mon[33453]: pgmap v603: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 23 MiB/s wr, 397 op/s 2024-09-21T02:15:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:21 smithi016 ceph-mon[31245]: pgmap v603: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 23 MiB/s wr, 397 op/s 2024-09-21T02:15:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:21 smithi161 ceph-mon[34061]: pgmap v603: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 23 MiB/s wr, 397 op/s 2024-09-21T02:15:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:23 smithi123 ceph-mon[33453]: pgmap v604: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 452 op/s 2024-09-21T02:15:24.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:23 smithi016 ceph-mon[31245]: pgmap v604: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 452 op/s 2024-09-21T02:15:24.324 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8fe5d51f', quiesce took 12.2 seconds. Will release after: 8.850000000000001 2024-09-21T02:15:24.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:23 smithi161 ceph-mon[34061]: pgmap v604: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 452 op/s 2024-09-21T02:15:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:25 smithi123 ceph-mon[33453]: pgmap v605: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 451 op/s 2024-09-21T02:15:26.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:25 smithi016 ceph-mon[31245]: pgmap v605: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 451 op/s 2024-09-21T02:15:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:25 smithi161 ceph-mon[34061]: pgmap v605: 129 pgs: 129 active+clean; 19 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 26 MiB/s wr, 451 op/s 2024-09-21T02:15:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:27 smithi123 ceph-mon[33453]: pgmap v606: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 25 MiB/s wr, 439 op/s 2024-09-21T02:15:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:27 smithi016 ceph-mon[31245]: pgmap v606: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 25 MiB/s wr, 439 op/s 2024-09-21T02:15:28.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:27 smithi161 ceph-mon[34061]: pgmap v606: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 25 MiB/s wr, 439 op/s 2024-09-21T02:15:30.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:29 smithi123 ceph-mon[33453]: pgmap v607: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 348 op/s 2024-09-21T02:15:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:29 smithi016 ceph-mon[31245]: pgmap v607: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 348 op/s 2024-09-21T02:15:30.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:29 smithi161 ceph-mon[34061]: pgmap v607: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 348 op/s 2024-09-21T02:15:32.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:31 smithi123 ceph-mon[33453]: pgmap v608: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 16 MiB/s wr, 287 op/s 2024-09-21T02:15:32.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:31 smithi016 ceph-mon[31245]: pgmap v608: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 16 MiB/s wr, 287 op/s 2024-09-21T02:15:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:31 smithi161 ceph-mon[34061]: pgmap v608: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 16 MiB/s wr, 287 op/s 2024-09-21T02:15:33.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:15:33.450 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 -- 172.21.15.16:0/1633346161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300109df0 msgr2=0x7fb300110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 --2- 172.21.15.16:0/1633346161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300109df0 0x7fb300110620 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009ca0 tx=0x7fb2f802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 -- 172.21.15.16:0/1633346161 shutdown_connections 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 --2- 172.21.15.16:0/1633346161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300109df0 0x7fb300110620 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 --2- 172.21.15.16:0/1633346161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 0x7fb3001098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 --2- 172.21.15.16:0/1633346161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300103b60 0x7fb300103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 -- 172.21.15.16:0/1633346161 >> 172.21.15.16:0/1633346161 conn(0x7fb30007b850 msgr2=0x7fb30007bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 -- 172.21.15.16:0/1633346161 shutdown_connections 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.943+0000 7fb305c3d640 1 -- 172.21.15.16:0/1633346161 wait complete. 2024-09-21T02:15:33.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 Processor -- start 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 -- start start 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 0x7fb30019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 0x7fb30019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001139b0 con 0x7fb300103b60 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb300113830 con 0x7fb3001094a0 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb305c3d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb300113b30 con 0x7fb300109df0 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.944+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45388/0 (socket says 172.21.15.16:45388) 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 -- 172.21.15.16:0/2038366495 learned_addr learned my addr 172.21.15.16:0/2038366495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2fffff640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 0x7fb30019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2feffd640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 0x7fb30019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 -- 172.21.15.16:0/2038366495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 msgr2=0x7fb30019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 0x7fb30019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 -- 172.21.15.16:0/2038366495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 msgr2=0x7fb30019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:33.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 0x7fb30019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 -- 172.21.15.16:0/2038366495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3001a0430 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2fffff640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 0x7fb30019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2ff7fe640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800e9f0 tx=0x7fb2e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800cd80 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3001a0720 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb2e80054b0 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8010430 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.945+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3001a6e80 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.947+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb300103f70 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.948+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fb2e80046c0 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.949+0000 7fb2fcff9640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 0x7fb2ec0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.949+0000 7fb2feffd640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 0x7fb2ec0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.949+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7fb2e80e80e0 con 0x7fb300103b60 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.949+0000 7fb2feffd640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 0x7fb2ec0b8b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb3001071a0 tx=0x7fb2f0005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:33.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:33.952+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb2e8056710 con 0x7fb300103b60 2024-09-21T02:15:34.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:33 smithi016 ceph-mon[31245]: pgmap v609: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 8.8 MiB/s wr, 158 op/s 2024-09-21T02:15:34.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.178+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb30019b5f0 con 0x7fb300103b60 2024-09-21T02:15:34.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.179+0000 7fb2fcff9640 1 -- 172.21.15.16:0/2038366495 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fb2e80ae280 con 0x7fb300103b60 2024-09-21T02:15:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:15:34.180 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:15:34.180 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 msgr2=0x7fb2ec0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 0x7fb2ec0b8b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb3001071a0 tx=0x7fb2f0005f90 comp rx=0 tx=0).stop 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 msgr2=0x7fb30019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800e9f0 tx=0x7fb2e800edf0 comp rx=0 tx=0).stop 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 shutdown_connections 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb2ec0b6680 0x7fb2ec0b8b90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb300109df0 0x7fb30019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3001094a0 0x7fb30019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 --2- 172.21.15.16:0/2038366495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb300103b60 0x7fb30019a4c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 >> 172.21.15.16:0/2038366495 conn(0x7fb30007b850 msgr2=0x7fb300105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 shutdown_connections 2024-09-21T02:15:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:34.181+0000 7fb305c3d640 1 -- 172.21.15.16:0/2038366495 wait complete. 2024-09-21T02:15:34.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:33 smithi123 ceph-mon[33453]: pgmap v609: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 8.8 MiB/s wr, 158 op/s 2024-09-21T02:15:34.416 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:33 smithi161 ceph-mon[34061]: pgmap v609: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 8.8 MiB/s wr, 158 op/s 2024-09-21T02:15:34.994 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 8fe5d51f --release --await' 2024-09-21T02:15:34.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 8fe5d51f --release --await 2024-09-21T02:15:35.262 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:15:35.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:34 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2038366495' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:35.316 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:34 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2038366495' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:35.383 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:34 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2038366495' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 -- 172.21.15.16:0/4079178219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109ec0 msgr2=0x7f305c1106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 --2- 172.21.15.16:0/4079178219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109ec0 0x7f305c1106f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3054009ca0 tx=0x7f305402a8c0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 -- 172.21.15.16:0/4079178219 shutdown_connections 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 --2- 172.21.15.16:0/4079178219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109ec0 0x7f305c1106f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 --2- 172.21.15.16:0/4079178219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c109570 0x7f305c109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 --2- 172.21.15.16:0/4079178219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c103c30 0x7f305c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 -- 172.21.15.16:0/4079178219 >> 172.21.15.16:0/4079178219 conn(0x7f305c07b8d0 msgr2=0x7f305c07bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 -- 172.21.15.16:0/4079178219 shutdown_connections 2024-09-21T02:15:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.822+0000 7f30643f0640 1 -- 172.21.15.16:0/4079178219 wait complete. 2024-09-21T02:15:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.823+0000 7f30643f0640 1 Processor -- start 2024-09-21T02:15:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.823+0000 7f30643f0640 1 -- start start 2024-09-21T02:15:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c103c30 0x7f305c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c109ec0 0x7f305c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1139e0 con 0x7f305c109570 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f305c113860 con 0x7f305c109ec0 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f30643f0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f305c113b60 con 0x7f305c103c30 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45408/0 (socket says 172.21.15.16:45408) 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 -- 172.21.15.16:0/2939414756 learned_addr learned my addr 172.21.15.16:0/2939414756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306218c640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c103c30 0x7f305c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306298d640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c109ec0 0x7f305c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 -- 172.21.15.16:0/2939414756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c103c30 msgr2=0x7f305c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c103c30 0x7f305c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 -- 172.21.15.16:0/2939414756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c109ec0 msgr2=0x7f305c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c109ec0 0x7f305c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 -- 172.21.15.16:0/2939414756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305c1a02c0 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.824+0000 7f306198b640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f304c00d750 tx=0x7f304c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.825+0000 7f30537fe640 1 -- 172.21.15.16:0/2939414756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c017070 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.825+0000 7f30537fe640 1 -- 172.21.15.16:0/2939414756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f304c0054b0 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.825+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305c1a05b0 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.825+0000 7f30537fe640 1 -- 172.21.15.16:0/2939414756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c004bc0 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.825+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305c1a6d10 con 0x7f305c109570 2024-09-21T02:15:35.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 Processor -- start 2024-09-21T02:15:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 -- start start 2024-09-21T02:15:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f30537fe640 1 -- 172.21.15.16:0/2939414756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f304c009c30 con 0x7f305c109570 2024-09-21T02:15:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce947ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cce9366dc0 con 0x55cce90b4840 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cce93f31f0 con 0x55cce90ac660 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.828+0000 7f306a964740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cce93e2620 con 0x55cce90d5520 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:53902/0 (socket says 172.21.15.16:53902) 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 -- 172.21.15.16:0/2764534114 learned_addr learned my addr 172.21.15.16:0/2764534114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306198b640 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306218c640 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce947ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 -- 172.21.15.16:0/2764534114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 msgr2=0x55cce947ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce947ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 -- 172.21.15.16:0/2764534114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 msgr2=0x55cce947bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 -- 172.21.15.16:0/2764534114 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cce93aafd0 con 0x55cce90d5520 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f30537fe640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 0x7f30480b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306298d640 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3054005cc0 tx=0x7f3054009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d64da936300d3991 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.829+0000 7f306218c640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 0x7f30480b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.830+0000 7f30537fe640 1 -- 172.21.15.16:0/2939414756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f304c0e7400 con 0x7f305c109570 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.830+0000 7f3050d9a640 1 -- 172.21.15.16:0/2764534114 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054031980 con 0x55cce90d5520 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.830+0000 7f3050d9a640 1 -- 172.21.15.16:0/2764534114 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3054036040 con 0x55cce90d5520 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.830+0000 7f3050d9a640 1 -- 172.21.15.16:0/2764534114 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054004310 con 0x55cce90d5520 2024-09-21T02:15:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- 172.21.15.16:0/2764534114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 msgr2=0x55cce947b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3054005cc0 tx=0x7f3054009a20 comp rx=0 tx=0).stop 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306218c640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 0x7f30480b8c10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55cce947a190 tx=0x7f305800b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- 172.21.15.16:0/2764534114 shutdown_connections 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce947ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 --2- 172.21.15.16:0/2764534114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce947b510 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- 172.21.15.16:0/2764534114 >> 172.21.15.16:0/2764534114 conn(0x55cce90c9910 msgr2=0x55cce9479400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- 172.21.15.16:0/2764534114 shutdown_connections 2024-09-21T02:15:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- 172.21.15.16:0/2764534114 wait complete. 2024-09-21T02:15:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.831+0000 7f306a964740 1 -- start start 2024-09-21T02:15:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 Processor -- start 2024-09-21T02:15:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce9486bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce9492e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cce937d300 con 0x55cce90b4840 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55cce93e6430 con 0x55cce90ac660 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55cce9243bb0 con 0x55cce90d5520 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306a964740 10 client.0 _ll_register_callbacks cb 0x55cce9397120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45426/0 (socket says 172.21.15.16:45426) 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306198b640 1 -- 172.21.15.16:0/2253090841 learned_addr learned my addr 172.21.15.16:0/2253090841 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306218c640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce9486bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.832+0000 7f306298d640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce9492e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 -- 172.21.15.16:0/2253090841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 msgr2=0x55cce9492e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce9492e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 -- 172.21.15.16:0/2253090841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 msgr2=0x55cce9487110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 -- 172.21.15.16:0/2253090841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cce936c780 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306198b640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306298d640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce9492e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306218c640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce9486bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3058005fd0 tx=0x7f3058004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4293505d20b61cc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f3045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058005020 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f3058017030 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.833+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3054036040 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3058032050 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 10 client.25706 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cce9140b00 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3054036040 con 0x55cce90ac660 2024-09-21T02:15:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f305801c600 con 0x55cce90ac660 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 10 client.25706 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 10 client.25706 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 20 client.25706 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce9093ea0 0x55cce9499040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 4 client.25706 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cce93f5140 con 0x55cce9093ea0 2024-09-21T02:15:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.834+0000 7f306198b640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce9093ea0 0x55cce9499040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.835+0000 7f306198b640 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce9093ea0 0x55cce9499040 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e85f7ccf887c02f9 server_cookie=bfc80b804db8e68e in_seq=0 out_seq=0 2024-09-21T02:15:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.835+0000 7f3045ffb640 10 client.25706 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f3045ffb640 1 -- 172.21.15.16:0/2253090841 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cce93f5140 con 0x55cce9093ea0 2024-09-21T02:15:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f3045ffb640 10 client.25706 handle_command_reply: tid=0 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 client.25706 shutdown 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 msgr2=0x55cce9486bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce9486bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3058005fd0 tx=0x7f3058004730 comp rx=0 tx=0).stop 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 shutdown_connections 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce9093ea0 0x55cce9499040 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90d5520 0x55cce9492e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce9487110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f3045ffb640 0 client.25706 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 --2- 172.21.15.16:0/2253090841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce90ac660 0x55cce9486bd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 >> 172.21.15.16:0/2253090841 conn(0x55cce90c9910 msgr2=0x55cce947e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 shutdown_connections 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 1 -- 172.21.15.16:0/2253090841 wait complete. 2024-09-21T02:15:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.836+0000 7f306a964740 20 client.25706 trim_cache size 0 max 16384 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.869+0000 7f306a964740 1 Processor -- start 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.869+0000 7f306a964740 1 -- start start 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce9488bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 0x55cce9492d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1b0ea0 con 0x55cce90b4840 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1b29e0 con 0x55cce9093ea0 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306a964740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1b1570 con 0x55cce90ac660 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45432/0 (socket says 172.21.15.16:45432) 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 -- 172.21.15.16:0/3300306211 learned_addr learned my addr 172.21.15.16:0/3300306211 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306198b640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 0x55cce9492d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306218c640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce9488bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 -- 172.21.15.16:0/3300306211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 msgr2=0x55cce9492d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 0x55cce9492d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 -- 172.21.15.16:0/3300306211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 msgr2=0x55cce9488bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce9488bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306298d640 1 -- 172.21.15.16:0/3300306211 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cce91b7960 con 0x55cce90b4840 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306198b640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 0x55cce9492d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.870+0000 7f306218c640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce9488bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.871+0000 7f306298d640 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f305402a900 tx=0x7f30540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb625e4e20d7f13 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.871+0000 7f3047fff640 1 -- 172.21.15.16:0/3300306211 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30540315d0 con 0x55cce90b4840 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.871+0000 7f3047fff640 1 -- 172.21.15.16:0/3300306211 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3054036070 con 0x55cce90b4840 2024-09-21T02:15:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.871+0000 7f3047fff640 1 -- 172.21.15.16:0/3300306211 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054004030 con 0x55cce90b4840 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 -- 172.21.15.16:0/3300306211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 msgr2=0x55cce947cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f305402a900 tx=0x7f30540095c0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 -- 172.21.15.16:0/3300306211 shutdown_connections 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90b4840 0x55cce947cf00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90ac660 0x55cce9492d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 --2- 172.21.15.16:0/3300306211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce9488bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 -- 172.21.15.16:0/3300306211 >> 172.21.15.16:0/3300306211 conn(0x55cce90c9910 msgr2=0x55cce9485450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 -- 172.21.15.16:0/3300306211 shutdown_connections 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.872+0000 7f306a964740 1 -- 172.21.15.16:0/3300306211 wait complete. 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 -- start start 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 Processor -- start 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce948b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90b4840 0x55cce9502a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1b2310 con 0x55cce90ac660 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1b1c40 con 0x55cce9093ea0 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f305c1aeea0 con 0x55cce90b4840 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306a964740 10 client.0 _ll_register_callbacks cb 0x7f305800bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306198b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45448/0 (socket says 172.21.15.16:45448) 2024-09-21T02:15:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.873+0000 7f306198b640 1 -- 172.21.15.16:0/1847011469 learned_addr learned my addr 172.21.15.16:0/1847011469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306298d640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90b4840 0x55cce9502a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306218c640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce948b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 -- 172.21.15.16:0/1847011469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90b4840 msgr2=0x55cce9502a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90b4840 0x55cce9502a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 -- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 msgr2=0x55cce948b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce948b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 -- 172.21.15.16:0/1847011469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cce9502f80 con 0x55cce90ac660 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306198b640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f304c0ace30 tx=0x7f304c094040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4d2f04399eaa server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f3045ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c020040 con 0x55cce90ac660 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9466+0+0 (secure 0 0 0) 0x7f304c0a9110 con 0x55cce90ac660 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.874+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3054036070 con 0x55cce90ac660 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f304c0ac7f0 con 0x55cce90ac660 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 10 client.16119 fetch_fsmap learned FSMap version 28 2024-09-21T02:15:35.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cce95040e0 con 0x55cce90ac660 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3054036070 con 0x55cce90ac660 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f304c0b1030 con 0x55cce90ac660 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 10 client.16119 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 10 client.16119 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 20 client.16119 populate_metadata read hostname 'smithi016' 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce90d5520 0x55cce9506850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 4 client.16119 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8fe5d51f", "release": true, "await": true}] 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8fe5d51f", "release": true, "await": true}) -- 0x55cce93f0bf0 con 0x55cce90d5520 2024-09-21T02:15:35.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.875+0000 7f306218c640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce90d5520 0x55cce9506850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:15:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.876+0000 7f3045ffb640 10 client.16119 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:35.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.876+0000 7f306218c640 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce90d5520 0x55cce9506850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f82a2ffad2c3203f server_cookie=b4ef84e9de42f06d in_seq=0 out_seq=0 2024-09-21T02:15:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f3045ffb640 1 -- 172.21.15.16:0/1847011469 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55cce93f0bf0 con 0x55cce90d5520 2024-09-21T02:15:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f3045ffb640 10 client.16119 handle_command_reply: tid=0 2024-09-21T02:15:35.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 client.16119 shutdown 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 msgr2=0x55cce948b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f304c0ace30 tx=0x7f304c094040 comp rx=0 tx=0).stop 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 shutdown_connections 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55cce90d5520 0x55cce9506850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cce90b4840 0x55cce9502a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cce90ac660 0x55cce948b6f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 --2- 172.21.15.16:0/1847011469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55cce9093ea0 0x55cce948b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 >> 172.21.15.16:0/1847011469 conn(0x55cce90c9910 msgr2=0x55cce9485e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.882+0000 7f3045ffb640 0 client.16119 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.883+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 shutdown_connections 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.883+0000 7f306a964740 1 -- 172.21.15.16:0/1847011469 wait complete. 2024-09-21T02:15:35.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.883+0000 7f306a964740 20 client.16119 trim_cache size 0 max 16384 2024-09-21T02:15:35.885 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:15:35.885 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:15:35.885 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:15:35.885 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 36, 2024-09-21T02:15:35.885 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "8fe5d51f": { 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 36, 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 141.1, 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:15:35.886 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 msgr2=0x7f30480b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 0x7f30480b8c10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55cce947a190 tx=0x7f305800b410 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 msgr2=0x7f305c19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f304c00d750 tx=0x7f304c00dc10 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 shutdown_connections 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f30480b6700 0x7f30480b8c10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f305c109ec0 0x7f305c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305c109570 0x7f305c19a8f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 --2- 172.21.15.16:0/2939414756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f305c103c30 0x7f305c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 >> 172.21.15.16:0/2939414756 conn(0x7f305c07b8d0 msgr2=0x7f305c107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 shutdown_connections 2024-09-21T02:15:35.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:15:35.884+0000 7f30643f0640 1 -- 172.21.15.16:0/2939414756 wait complete. 2024-09-21T02:15:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:35 smithi016 ceph-mon[31245]: pgmap v610: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 4.8 MiB/s wr, 93 op/s 2024-09-21T02:15:36.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:35 smithi123 ceph-mon[33453]: pgmap v610: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 4.8 MiB/s wr, 93 op/s 2024-09-21T02:15:36.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:35 smithi161 ceph-mon[34061]: pgmap v610: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 4.8 MiB/s wr, 93 op/s 2024-09-21T02:15:36.757 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8fe5d51f', total seconds elapsed: 24.6 2024-09-21T02:15:38.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:37 smithi123 ceph-mon[33453]: pgmap v611: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 749 KiB/s wr, 20 op/s 2024-09-21T02:15:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:37 smithi016 ceph-mon[31245]: pgmap v611: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 749 KiB/s wr, 20 op/s 2024-09-21T02:15:38.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:37 smithi161 ceph-mon[34061]: pgmap v611: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 749 KiB/s wr, 20 op/s 2024-09-21T02:15:39.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:38 smithi123 ceph-mon[33453]: pgmap v612: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.5 MiB/s wr, 35 op/s 2024-09-21T02:15:39.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:38 smithi016 ceph-mon[31245]: pgmap v612: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.5 MiB/s wr, 35 op/s 2024-09-21T02:15:39.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:38 smithi161 ceph-mon[34061]: pgmap v612: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.5 MiB/s wr, 35 op/s 2024-09-21T02:15:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:41 smithi016 ceph-mon[31245]: pgmap v613: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 6.1 MiB/s wr, 117 op/s 2024-09-21T02:15:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:41 smithi161 ceph-mon[34061]: pgmap v613: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 6.1 MiB/s wr, 117 op/s 2024-09-21T02:15:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:41 smithi123 ceph-mon[33453]: pgmap v613: 129 pgs: 129 active+clean; 19 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 6.1 MiB/s wr, 117 op/s 2024-09-21T02:15:43.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:43 smithi016 ceph-mon[31245]: pgmap v614: 129 pgs: 129 active+clean; 19 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 11 MiB/s wr, 192 op/s 2024-09-21T02:15:43.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:43 smithi161 ceph-mon[34061]: pgmap v614: 129 pgs: 129 active+clean; 19 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 11 MiB/s wr, 192 op/s 2024-09-21T02:15:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:43 smithi123 ceph-mon[33453]: pgmap v614: 129 pgs: 129 active+clean; 19 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 11 MiB/s wr, 192 op/s 2024-09-21T02:15:45.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:45 smithi016 ceph-mon[31245]: pgmap v615: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 381 op/s 2024-09-21T02:15:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:45 smithi161 ceph-mon[34061]: pgmap v615: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 381 op/s 2024-09-21T02:15:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:45 smithi123 ceph-mon[33453]: pgmap v615: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 21 MiB/s wr, 381 op/s 2024-09-21T02:15:46.632 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:46 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:15:46.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:46 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:15:46.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:46 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:15:47.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:47 smithi016 ceph-mon[31245]: pgmap v616: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 417 op/s 2024-09-21T02:15:47.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:15:47.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:15:47.661 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:15:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:47 smithi161 ceph-mon[34061]: pgmap v616: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 417 op/s 2024-09-21T02:15:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:15:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:15:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:15:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:47 smithi123 ceph-mon[33453]: pgmap v616: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 417 op/s 2024-09-21T02:15:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:15:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:15:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:15:49.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:49 smithi016 ceph-mon[31245]: pgmap v617: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 28 MiB/s wr, 499 op/s 2024-09-21T02:15:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:49 smithi161 ceph-mon[34061]: pgmap v617: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 28 MiB/s wr, 499 op/s 2024-09-21T02:15:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:49 smithi123 ceph-mon[33453]: pgmap v617: 129 pgs: 129 active+clean; 19 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 28 MiB/s wr, 499 op/s 2024-09-21T02:15:51.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:51 smithi016 ceph-mon[31245]: pgmap v618: 129 pgs: 129 active+clean; 19 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 35 MiB/s wr, 624 op/s 2024-09-21T02:15:51.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:51 smithi161 ceph-mon[34061]: pgmap v618: 129 pgs: 129 active+clean; 19 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 35 MiB/s wr, 624 op/s 2024-09-21T02:15:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:51 smithi123 ceph-mon[33453]: pgmap v618: 129 pgs: 129 active+clean; 19 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 35 MiB/s wr, 624 op/s 2024-09-21T02:15:53.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:53 smithi016 ceph-mon[31245]: pgmap v619: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 35 MiB/s wr, 619 op/s 2024-09-21T02:15:53.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:53 smithi161 ceph-mon[34061]: pgmap v619: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 35 MiB/s wr, 619 op/s 2024-09-21T02:15:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:53 smithi123 ceph-mon[33453]: pgmap v619: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 35 MiB/s wr, 619 op/s 2024-09-21T02:15:55.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:55 smithi016 ceph-mon[31245]: pgmap v620: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 42 MiB/s wr, 729 op/s 2024-09-21T02:15:55.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:55 smithi161 ceph-mon[34061]: pgmap v620: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 42 MiB/s wr, 729 op/s 2024-09-21T02:15:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:55 smithi123 ceph-mon[33453]: pgmap v620: 129 pgs: 129 active+clean; 19 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 42 MiB/s wr, 729 op/s 2024-09-21T02:15:57.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:57 smithi016 ceph-mon[31245]: pgmap v621: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 33 MiB/s wr, 580 op/s 2024-09-21T02:15:57.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:57 smithi161 ceph-mon[34061]: pgmap v621: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 33 MiB/s wr, 580 op/s 2024-09-21T02:15:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:57 smithi123 ceph-mon[33453]: pgmap v621: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 33 MiB/s wr, 580 op/s 2024-09-21T02:15:59.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:15:59 smithi161 ceph-mon[34061]: pgmap v622: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 36 MiB/s wr, 628 op/s 2024-09-21T02:15:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:15:59 smithi123 ceph-mon[33453]: pgmap v622: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 36 MiB/s wr, 628 op/s 2024-09-21T02:15:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:15:59 smithi016 ceph-mon[31245]: pgmap v622: 129 pgs: 129 active+clean; 19 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 36 MiB/s wr, 628 op/s 2024-09-21T02:16:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:01 smithi123 ceph-mon[33453]: pgmap v623: 129 pgs: 129 active+clean; 19 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 39 MiB/s wr, 696 op/s 2024-09-21T02:16:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:01 smithi016 ceph-mon[31245]: pgmap v623: 129 pgs: 129 active+clean; 19 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 39 MiB/s wr, 696 op/s 2024-09-21T02:16:02.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:01 smithi161 ceph-mon[34061]: pgmap v623: 129 pgs: 129 active+clean; 19 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 39 MiB/s wr, 696 op/s 2024-09-21T02:16:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:03 smithi123 ceph-mon[33453]: pgmap v624: 129 pgs: 129 active+clean; 20 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 36 MiB/s wr, 634 op/s 2024-09-21T02:16:04.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:03 smithi016 ceph-mon[31245]: pgmap v624: 129 pgs: 129 active+clean; 20 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 36 MiB/s wr, 634 op/s 2024-09-21T02:16:04.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:03 smithi161 ceph-mon[34061]: pgmap v624: 129 pgs: 129 active+clean; 20 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 36 MiB/s wr, 634 op/s 2024-09-21T02:16:06.217 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:05 smithi161 ceph-mon[34061]: pgmap v625: 129 pgs: 129 active+clean; 20 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 41 MiB/s wr, 754 op/s 2024-09-21T02:16:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:05 smithi123 ceph-mon[33453]: pgmap v625: 129 pgs: 129 active+clean; 20 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 41 MiB/s wr, 754 op/s 2024-09-21T02:16:06.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:05 smithi016 ceph-mon[31245]: pgmap v625: 129 pgs: 129 active+clean; 20 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 41 MiB/s wr, 754 op/s 2024-09-21T02:16:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:07 smithi123 ceph-mon[33453]: pgmap v626: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 32 MiB/s wr, 601 op/s 2024-09-21T02:16:08.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:07 smithi016 ceph-mon[31245]: pgmap v626: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 32 MiB/s wr, 601 op/s 2024-09-21T02:16:08.414 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:07 smithi161 ceph-mon[34061]: pgmap v626: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 32 MiB/s wr, 601 op/s 2024-09-21T02:16:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:09 smithi123 ceph-mon[33453]: pgmap v627: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 35 MiB/s wr, 639 op/s 2024-09-21T02:16:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:09 smithi016 ceph-mon[31245]: pgmap v627: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 35 MiB/s wr, 639 op/s 2024-09-21T02:16:10.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:09 smithi161 ceph-mon[34061]: pgmap v627: 129 pgs: 129 active+clean; 20 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 35 MiB/s wr, 639 op/s 2024-09-21T02:16:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:11 smithi123 ceph-mon[33453]: pgmap v628: 129 pgs: 129 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 37 MiB/s wr, 692 op/s 2024-09-21T02:16:12.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:11 smithi123 ceph-mon[33453]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T02:16:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:11 smithi016 ceph-mon[31245]: pgmap v628: 129 pgs: 129 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 37 MiB/s wr, 692 op/s 2024-09-21T02:16:12.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:11 smithi016 ceph-mon[31245]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T02:16:12.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:11 smithi161 ceph-mon[34061]: pgmap v628: 129 pgs: 129 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 37 MiB/s wr, 692 op/s 2024-09-21T02:16:12.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:11 smithi161 ceph-mon[34061]: osdmap e92: 12 total, 12 up, 12 in 2024-09-21T02:16:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:13 smithi123 ceph-mon[33453]: pgmap v630: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 34 MiB/s wr, 641 op/s 2024-09-21T02:16:14.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:13 smithi016 ceph-mon[31245]: pgmap v630: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 34 MiB/s wr, 641 op/s 2024-09-21T02:16:14.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:13 smithi161 ceph-mon[34061]: pgmap v630: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 34 MiB/s wr, 641 op/s 2024-09-21T02:16:15.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:14 smithi016 ceph-mon[31245]: pgmap v631: 129 pgs: 21 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 54 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 33 MiB/s wr, 598 op/s 2024-09-21T02:16:15.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:14 smithi123 ceph-mon[33453]: pgmap v631: 129 pgs: 21 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 54 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 33 MiB/s wr, 598 op/s 2024-09-21T02:16:15.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:14 smithi161 ceph-mon[34061]: pgmap v631: 129 pgs: 21 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 54 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 33 MiB/s wr, 598 op/s 2024-09-21T02:16:17.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:17 smithi016 ceph-mon[31245]: pgmap v632: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 31 MiB/s wr, 575 op/s 2024-09-21T02:16:17.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:17 smithi161 ceph-mon[34061]: pgmap v632: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 31 MiB/s wr, 575 op/s 2024-09-21T02:16:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:17 smithi123 ceph-mon[33453]: pgmap v632: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 20 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 31 MiB/s wr, 575 op/s 2024-09-21T02:16:19.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:19 smithi016 ceph-mon[31245]: pgmap v633: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 31 MiB/s wr, 578 op/s 2024-09-21T02:16:19.664 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:19 smithi161 ceph-mon[34061]: pgmap v633: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 31 MiB/s wr, 578 op/s 2024-09-21T02:16:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:19 smithi123 ceph-mon[33453]: pgmap v633: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 31 MiB/s wr, 578 op/s 2024-09-21T02:16:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:21 smithi123 ceph-mon[33453]: pgmap v634: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:21 smithi123 ceph-mon[33453]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T02:16:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:21 smithi016 ceph-mon[31245]: pgmap v634: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:22.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:21 smithi016 ceph-mon[31245]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T02:16:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:21 smithi161 ceph-mon[34061]: pgmap v634: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:21 smithi161 ceph-mon[34061]: osdmap e93: 12 total, 12 up, 12 in 2024-09-21T02:16:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:23 smithi123 ceph-mon[33453]: pgmap v636: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:24.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:23 smithi016 ceph-mon[31245]: pgmap v636: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:24.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:23 smithi161 ceph-mon[34061]: pgmap v636: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 20 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 30 MiB/s wr, 558 op/s 2024-09-21T02:16:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:25 smithi123 ceph-mon[33453]: pgmap v637: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 581 op/s 2024-09-21T02:16:26.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:25 smithi016 ceph-mon[31245]: pgmap v637: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 581 op/s 2024-09-21T02:16:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:25 smithi161 ceph-mon[34061]: pgmap v637: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 581 op/s 2024-09-21T02:16:27.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:26 smithi123 ceph-mon[33453]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T02:16:27.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:26 smithi016 ceph-mon[31245]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T02:16:27.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:26 smithi161 ceph-mon[34061]: osdmap e94: 12 total, 12 up, 12 in 2024-09-21T02:16:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:27 smithi123 ceph-mon[33453]: pgmap v639: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 32 MiB/s wr, 613 op/s 2024-09-21T02:16:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:27 smithi016 ceph-mon[31245]: pgmap v639: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 32 MiB/s wr, 613 op/s 2024-09-21T02:16:28.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:27 smithi161 ceph-mon[34061]: pgmap v639: 129 pgs: 129 active+clean; 20 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 32 MiB/s wr, 613 op/s 2024-09-21T02:16:30.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:29 smithi123 ceph-mon[33453]: pgmap v640: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 577 op/s 2024-09-21T02:16:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:29 smithi016 ceph-mon[31245]: pgmap v640: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 577 op/s 2024-09-21T02:16:30.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:29 smithi161 ceph-mon[34061]: pgmap v640: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 31 MiB/s wr, 577 op/s 2024-09-21T02:16:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:31 smithi016 ceph-mon[31245]: pgmap v641: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:31.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:31 smithi161 ceph-mon[34061]: pgmap v641: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:31.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:31 smithi123 ceph-mon[33453]: pgmap v641: 129 pgs: 129 active+clean; 20 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:33 smithi016 ceph-mon[31245]: pgmap v642: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 34 MiB/s wr, 644 op/s 2024-09-21T02:16:33.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:33 smithi161 ceph-mon[34061]: pgmap v642: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 34 MiB/s wr, 644 op/s 2024-09-21T02:16:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:33 smithi123 ceph-mon[33453]: pgmap v642: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 34 MiB/s wr, 644 op/s 2024-09-21T02:16:35.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:35 smithi016 ceph-mon[31245]: pgmap v643: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 34 MiB/s wr, 654 op/s 2024-09-21T02:16:35.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:35 smithi161 ceph-mon[34061]: pgmap v643: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 34 MiB/s wr, 654 op/s 2024-09-21T02:16:35.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:35 smithi123 ceph-mon[33453]: pgmap v643: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 34 MiB/s wr, 654 op/s 2024-09-21T02:16:37.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:37 smithi016 ceph-mon[31245]: pgmap v644: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:37.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:37 smithi161 ceph-mon[34061]: pgmap v644: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:37.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:37 smithi123 ceph-mon[33453]: pgmap v644: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 36 MiB/s wr, 683 op/s 2024-09-21T02:16:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:39 smithi016 ceph-mon[31245]: pgmap v645: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 33 MiB/s wr, 631 op/s 2024-09-21T02:16:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:39 smithi161 ceph-mon[34061]: pgmap v645: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 33 MiB/s wr, 631 op/s 2024-09-21T02:16:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:39 smithi123 ceph-mon[33453]: pgmap v645: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 33 MiB/s wr, 631 op/s 2024-09-21T02:16:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:41 smithi016 ceph-mon[31245]: pgmap v646: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 35 MiB/s wr, 680 op/s 2024-09-21T02:16:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:41 smithi161 ceph-mon[34061]: pgmap v646: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 35 MiB/s wr, 680 op/s 2024-09-21T02:16:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:41 smithi123 ceph-mon[33453]: pgmap v646: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 35 MiB/s wr, 680 op/s 2024-09-21T02:16:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:43 smithi016 ceph-mon[31245]: pgmap v647: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:43 smithi161 ceph-mon[34061]: pgmap v647: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:43 smithi123 ceph-mon[33453]: pgmap v647: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:45 smithi016 ceph-mon[31245]: pgmap v648: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 37 MiB/s wr, 728 op/s 2024-09-21T02:16:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:45 smithi161 ceph-mon[34061]: pgmap v648: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 37 MiB/s wr, 728 op/s 2024-09-21T02:16:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:45 smithi123 ceph-mon[33453]: pgmap v648: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 37 MiB/s wr, 728 op/s 2024-09-21T02:16:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:47 smithi161 ceph-mon[34061]: pgmap v649: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 30 MiB/s wr, 593 op/s 2024-09-21T02:16:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:47 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:16:47.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:47 smithi123 ceph-mon[33453]: pgmap v649: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 30 MiB/s wr, 593 op/s 2024-09-21T02:16:47.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:47 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:16:47.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:47 smithi016 ceph-mon[31245]: pgmap v649: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 30 MiB/s wr, 593 op/s 2024-09-21T02:16:47.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:47 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:16:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:49 smithi016 ceph-mon[31245]: pgmap v650: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 32 MiB/s wr, 619 op/s 2024-09-21T02:16:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:49 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:49 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:49 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:49 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:49 smithi161 ceph-mon[34061]: pgmap v650: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 32 MiB/s wr, 619 op/s 2024-09-21T02:16:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:49 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:49 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:49 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:49 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:50.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:49 smithi123 ceph-mon[33453]: pgmap v650: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 32 MiB/s wr, 619 op/s 2024-09-21T02:16:50.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:49 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:50.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:49 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:50.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:49 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:50.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:49 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: pgmap v651: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 35 MiB/s wr, 691 op/s 2024-09-21T02:16:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:16:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:16:51.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:51 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: pgmap v651: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 35 MiB/s wr, 691 op/s 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:16:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:51 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: pgmap v651: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 35 MiB/s wr, 691 op/s 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:16:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:51 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:16:53.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:53 smithi123 ceph-mon[33453]: pgmap v652: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:53.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:53 smithi016 ceph-mon[31245]: pgmap v652: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:53.914 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:53 smithi161 ceph-mon[34061]: pgmap v652: 129 pgs: 129 active+clean; 20 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 33 MiB/s wr, 646 op/s 2024-09-21T02:16:54.944 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-21T02:16:54.944 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T02:16:54.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:16:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:16:55.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:55 smithi123 ceph-mon[33453]: pgmap v653: 129 pgs: 129 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 36 MiB/s wr, 691 op/s 2024-09-21T02:16:55.791 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:55 smithi016 ceph-mon[31245]: pgmap v653: 129 pgs: 129 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 36 MiB/s wr, 691 op/s 2024-09-21T02:16:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:55 smithi161 ceph-mon[34061]: pgmap v653: 129 pgs: 129 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 36 MiB/s wr, 691 op/s 2024-09-21T02:16:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.367+0000 7f3e23fff640 1 -- 172.21.15.16:0/3824824009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e241081b0 msgr2=0x7f3e241085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:16:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.367+0000 7f3e23fff640 1 --2- 172.21.15.16:0/3824824009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e241081b0 0x7f3e241085c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009ca0 tx=0x7f3e1c02a8d0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 -- 172.21.15.16:0/3824824009 shutdown_connections 2024-09-21T02:16:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 --2- 172.21.15.16:0/3824824009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e24108b00 0x7f3e2410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 --2- 172.21.15.16:0/3824824009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e241081b0 0x7f3e241085c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 --2- 172.21.15.16:0/3824824009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 -- 172.21.15.16:0/3824824009 >> 172.21.15.16:0/3824824009 conn(0x7f3e240fe2c0 msgr2=0x7f3e24100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 -- 172.21.15.16:0/3824824009 shutdown_connections 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.368+0000 7f3e23fff640 1 -- 172.21.15.16:0/3824824009 wait complete. 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 Processor -- start 2024-09-21T02:16:56.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 -- start start 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 0x7f3e24199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 0x7f3e2419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24113760 con 0x7f3e24102870 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241135e0 con 0x7f3e241081b0 2024-09-21T02:16:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.369+0000 7f3e23fff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241138e0 con 0x7f3e24108b00 2024-09-21T02:16:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:16:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47576/0 (socket says 172.21.15.16:47576) 2024-09-21T02:16:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e22ffd640 1 -- 172.21.15.16:0/843805826 learned_addr learned my addr 172.21.15.16:0/843805826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:16:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 0x7f3e24199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e237fe640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 0x7f3e2419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 -- 172.21.15.16:0/843805826 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 msgr2=0x7f3e2419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 0x7f3e2419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 -- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 msgr2=0x7f3e24198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e227fc640 1 -- 172.21.15.16:0/843805826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241a01d0 con 0x7f3e241081b0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.370+0000 7f3e22ffd640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.371+0000 7f3e237fe640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 0x7f3e2419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.371+0000 7f3e227fc640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 0x7f3e24199170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02adf0 tx=0x7f3e1c0305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.371+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c030bd0 con 0x7f3e241081b0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.372+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e1c030d70 con 0x7f3e241081b0 2024-09-21T02:16:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.372+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c03a660 con 0x7f3e241081b0 2024-09-21T02:16:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.372+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241a0460 con 0x7f3e241081b0 2024-09-21T02:16:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.372+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a0920 con 0x7f3e241081b0 2024-09-21T02:16:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.375+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e24102c80 con 0x7f3e241081b0 2024-09-21T02:16:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.376+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3e1c03a800 con 0x7f3e241081b0 2024-09-21T02:16:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.377+0000 7f3e0bfff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 0x7f3e0c0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:16:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.378+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3e1c108d20 con 0x7f3e241081b0 2024-09-21T02:16:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.383+0000 7f3e22ffd640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 0x7f3e0c0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:16:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.385+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3e1c0cee60 con 0x7f3e241081b0 2024-09-21T02:16:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.391+0000 7f3e22ffd640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 0x7f3e0c0b8d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e14005fd0 tx=0x7f3e1400b480 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:16:56.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.845+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e24100860 con 0x7f3e241081b0 2024-09-21T02:16:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:16:56.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:16:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:16:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.853+0000 7f3e0bfff640 1 -- 172.21.15.16:0/843805826 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f3e1c0d3240 con 0x7f3e241081b0 2024-09-21T02:16:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 msgr2=0x7f3e0c0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 0x7f3e0c0b8d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e14005fd0 tx=0x7f3e1400b480 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 msgr2=0x7f3e24199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 0x7f3e24199170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c02adf0 tx=0x7f3e1c0305d0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 shutdown_connections 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f3e0c0b6850 0x7f3e0c0b8d60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e24108b00 0x7f3e2419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e241081b0 0x7f3e24199170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 --2- 172.21.15.16:0/843805826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e24102870 0x7f3e24198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.857+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 >> 172.21.15.16:0/843805826 conn(0x7f3e240fe2c0 msgr2=0x7f3e241078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.858+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 shutdown_connections 2024-09-21T02:16:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:16:56.858+0000 7f3e23fff640 1 -- 172.21.15.16:0/843805826 wait complete. 2024-09-21T02:16:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:57 smithi123 ceph-mon[33453]: pgmap v654: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 30 MiB/s wr, 585 op/s 2024-09-21T02:16:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:57 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/843805826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:16:57.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:57 smithi016 ceph-mon[31245]: pgmap v654: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 30 MiB/s wr, 585 op/s 2024-09-21T02:16:57.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:57 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/843805826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:16:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:57 smithi161 ceph-mon[34061]: pgmap v654: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 30 MiB/s wr, 585 op/s 2024-09-21T02:16:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:57 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/843805826' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:16:59.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:16:59 smithi016 ceph-mon[31245]: pgmap v655: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 31 MiB/s wr, 596 op/s 2024-09-21T02:16:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:16:59 smithi123 ceph-mon[33453]: pgmap v655: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 31 MiB/s wr, 596 op/s 2024-09-21T02:16:59.872 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T02:16:59.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T02:16:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:16:59 smithi161 ceph-mon[34061]: pgmap v655: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 31 MiB/s wr, 596 op/s 2024-09-21T02:17:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.281+0000 7fa5997ef640 1 -- 172.21.15.16:0/1376568867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410f160 msgr2=0x7fa59410f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.281+0000 7fa5997ef640 1 --2- 172.21.15.16:0/1376568867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410f160 0x7fa59410f570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa58400af00 tx=0x7fa58402e630 comp rx=0 tx=0).stop 2024-09-21T02:17:01.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.282+0000 7fa5997ef640 1 -- 172.21.15.16:0/1376568867 shutdown_connections 2024-09-21T02:17:01.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.282+0000 7fa5997ef640 1 --2- 172.21.15.16:0/1376568867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa594116fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.282+0000 7fa5997ef640 1 --2- 172.21.15.16:0/1376568867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa59410a3c0 0x7fa59410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.282+0000 7fa5997ef640 1 --2- 172.21.15.16:0/1376568867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410f160 0x7fa59410f570 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa58400af00 tx=0x7fa58402e630 comp rx=0 tx=0).stop 2024-09-21T02:17:01.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.282+0000 7fa5997ef640 1 -- 172.21.15.16:0/1376568867 >> 172.21.15.16:0/1376568867 conn(0x7fa5940712b0 msgr2=0x7fa594073e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.285+0000 7fa5997ef640 1 -- 172.21.15.16:0/1376568867 shutdown_connections 2024-09-21T02:17:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.286+0000 7fa5997ef640 1 -- 172.21.15.16:0/1376568867 wait complete. 2024-09-21T02:17:01.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.288+0000 7fa5997ef640 1 Processor -- start 2024-09-21T02:17:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 -- start start 2024-09-21T02:17:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410a3c0 0x7fa5941a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941aec00 0x7fa5941af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa59411a060 con 0x7fa5941aec00 2024-09-21T02:17:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa594119ee0 con 0x7fa59410ad10 2024-09-21T02:17:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.292+0000 7fa5997ef640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa59411a1e0 con 0x7fa59410a3c0 2024-09-21T02:17:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.293+0000 7fa58bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.293+0000 7fa58bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:40002/0 (socket says 172.21.15.16:40002) 2024-09-21T02:17:01.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.293+0000 7fa58bfff640 1 -- 172.21.15.16:0/2016716952 learned_addr learned my addr 172.21.15.16:0/2016716952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.294+0000 7fa58bfff640 1 -- 172.21.15.16:0/2016716952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410a3c0 msgr2=0x7fa5941a9480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.294+0000 7fa58bfff640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410a3c0 0x7fa5941a9480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.294+0000 7fa58bfff640 1 -- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941aec00 msgr2=0x7fa5941af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.294+0000 7fa58bfff640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941aec00 0x7fa5941af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.294+0000 7fa58bfff640 1 -- 172.21.15.16:0/2016716952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5941af7e0 con 0x7fa59410ad10 2024-09-21T02:17:01.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.295+0000 7fa58bfff640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa57c006fd0 tx=0x7fa57c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.298+0000 7fa590ff9640 1 -- 172.21.15.16:0/2016716952 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c011830 con 0x7fa59410ad10 2024-09-21T02:17:01.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.298+0000 7fa590ff9640 1 -- 172.21.15.16:0/2016716952 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa57c016070 con 0x7fa59410ad10 2024-09-21T02:17:01.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.298+0000 7fa590ff9640 1 -- 172.21.15.16:0/2016716952 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c010d60 con 0x7fa59410ad10 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.300+0000 7fa5997ef640 1 -- 172.21.15.16:0/2016716952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5941b5d80 con 0x7fa59410ad10 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.300+0000 7fa5997ef640 1 -- 172.21.15.16:0/2016716952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5941b6200 con 0x7fa59410ad10 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.303+0000 7fa59fd63740 1 Processor -- start 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 -- start start 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099da980 0x55ec09daae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099e2b60 0x55ec09dab380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ec09d20b90 con 0x55ec099e2b60 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ec09d0ffc0 con 0x55ec099da980 2024-09-21T02:17:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.304+0000 7fa59fd63740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ec09d13dd0 con 0x55ec09a03840 2024-09-21T02:17:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.307+0000 7fa590ff9640 1 -- 172.21.15.16:0/2016716952 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fa57c01b460 con 0x7fa59410ad10 2024-09-21T02:17:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa592ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099da980 0x55ec09daae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:48702/0 (socket says 172.21.15.16:48702) 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 -- 172.21.15.16:0/3337587082 learned_addr learned my addr 172.21.15.16:0/3337587082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa590ff9640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 0x7fa5800b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 -- 172.21.15.16:0/3337587082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099da980 msgr2=0x55ec09daae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099da980 0x55ec09daae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 -- 172.21.15.16:0/3337587082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099e2b60 msgr2=0x55ec09dab380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099e2b60 0x55ec09dab380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.308+0000 7fa5937fe640 1 -- 172.21.15.16:0/3337587082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec09c97060 con 0x55ec09a03840 2024-09-21T02:17:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.309+0000 7fa590ff9640 1 -- 172.21.15.16:0/2016716952 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa57c0e7010 con 0x7fa59410ad10 2024-09-21T02:17:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.309+0000 7fa5937fe640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 0x7fa5800b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.310+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009ee0 tx=0x7fa58c00b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316bed3703966ce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.312+0000 7fa5937fe640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 0x7fa5800b8d90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa58c01d000 tx=0x7fa58c01a000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.313+0000 7fa588ff9640 1 -- 172.21.15.16:0/3337587082 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c01f8b0 con 0x55ec09a03840 2024-09-21T02:17:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.313+0000 7fa588ff9640 1 -- 172.21.15.16:0/3337587082 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa58c020480 con 0x55ec09a03840 2024-09-21T02:17:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 -- 172.21.15.16:0/3337587082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 msgr2=0x55ec09daa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009ee0 tx=0x7fa58c00b420 comp rx=0 tx=0).stop 2024-09-21T02:17:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 -- 172.21.15.16:0/3337587082 shutdown_connections 2024-09-21T02:17:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099e2b60 0x55ec09dab380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099da980 0x55ec09daae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3337587082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec09a03840 0x55ec09daa900 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.314+0000 7fa59fd63740 1 -- 172.21.15.16:0/3337587082 >> 172.21.15.16:0/3337587082 conn(0x55ec099f7990 msgr2=0x55ec09da87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.317+0000 7fa59fd63740 1 -- 172.21.15.16:0/3337587082 shutdown_connections 2024-09-21T02:17:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.317+0000 7fa59fd63740 1 -- 172.21.15.16:0/3337587082 wait complete. 2024-09-21T02:17:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.317+0000 7fa59fd63740 1 -- start start 2024-09-21T02:17:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.317+0000 7fa59fd63740 1 Processor -- start 2024-09-21T02:17:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09db60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09db6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09de43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ec09b717a0 con 0x55ec099da980 2024-09-21T02:17:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ec09cb50a0 con 0x55ec09a03840 2024-09-21T02:17:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55ec09ddb060 con 0x55ec099e2b60 2024-09-21T02:17:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.318+0000 7fa59fd63740 10 client.0 _ll_register_callbacks cb 0x55ec09c923c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.319+0000 7fa58bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09db6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.319+0000 7fa58bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09db6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:48716/0 (socket says 172.21.15.16:48716) 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.319+0000 7fa58bfff640 1 -- 172.21.15.16:0/1868560478 learned_addr learned my addr 172.21.15.16:0/1868560478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.319+0000 7fa5937fe640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09de43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.322+0000 7fa5937fe640 1 -- 172.21.15.16:0/1868560478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 msgr2=0x55ec09db6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.322+0000 7fa5937fe640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09db6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.322+0000 7fa5937fe640 1 -- 172.21.15.16:0/1868560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 msgr2=0x55ec09db60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.322+0000 7fa5937fe640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09db60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.322+0000 7fa5937fe640 1 -- 172.21.15.16:0/1868560478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec09ca3880 con 0x55ec09a03840 2024-09-21T02:17:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.326+0000 7fa5937fe640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09de43a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009ee0 tx=0x7fa58c01fae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be754526904bc2d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.326+0000 7fa57a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-21T02:17:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.326+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c025040 con 0x55ec09a03840 2024-09-21T02:17:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.327+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mon.2 v2:172.21.15.123:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa58c01fdd0 con 0x55ec09a03840 2024-09-21T02:17:01.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.327+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa58c020480 con 0x55ec09a03840 2024-09-21T02:17:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.328+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa58c01e850 con 0x55ec09a03840 2024-09-21T02:17:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.329+0000 7fa59fd63740 10 client.25724 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:01.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.329+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec09a7db90 con 0x55ec09a03840 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.330+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mon.2 v2:172.21.15.123:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa58c020480 con 0x55ec09a03840 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.330+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mon.2 v2:172.21.15.123:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa58c028030 con 0x55ec09a03840 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 10 client.25724 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 10 client.25724 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 20 client.25724 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:01.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec099c21c0 0x55ec09dea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 4 client.25724 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:17:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.333+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec09d22ae0 con 0x55ec099c21c0 2024-09-21T02:17:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.339+0000 7fa592ffd640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec099c21c0 0x55ec09dea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.348+0000 7fa592ffd640 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec099c21c0 0x55ec09dea1f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff51c4e2a566eaa6 server_cookie=db30dc2f5ccf658e in_seq=0 out_seq=0 2024-09-21T02:17:01.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.349+0000 7fa57a7fc640 10 client.25724 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.350+0000 7fa57a7fc640 1 -- 172.21.15.16:0/1868560478 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ec09d22ae0 con 0x55ec099c21c0 2024-09-21T02:17:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.350+0000 7fa57a7fc640 10 client.25724 handle_command_reply: tid=0 2024-09-21T02:17:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.352+0000 7fa59fd63740 1 client.25724 shutdown 2024-09-21T02:17:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 msgr2=0x55ec09de43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09de43a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009ee0 tx=0x7fa58c01fae0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 shutdown_connections 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec099c21c0 0x55ec09dea1f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09de43a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09db6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 --2- 172.21.15.16:0/1868560478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09db60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa57a7fc640 0 client.25724 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.356+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 >> 172.21.15.16:0/1868560478 conn(0x55ec099f7990 msgr2=0x55ec09dadae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.357+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 shutdown_connections 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.357+0000 7fa59fd63740 1 -- 172.21.15.16:0/1868560478 wait complete. 2024-09-21T02:17:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.358+0000 7fa59fd63740 20 client.25724 trim_cache size 0 max 16384 2024-09-21T02:17:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.388+0000 7fa59fd63740 1 Processor -- start 2024-09-21T02:17:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.388+0000 7fa59fd63740 1 -- start start 2024-09-21T02:17:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09de47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09def8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa594076fa0 con 0x55ec099da980 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa594078ae0 con 0x55ec09a03840 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa59fd63740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa594077670 con 0x55ec099e2b60 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49582/0 (socket says 172.21.15.16:49582) 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 -- 172.21.15.16:0/3668839989 learned_addr learned my addr 172.21.15.16:0/3668839989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa58bfff640 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09de47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09def8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 -- 172.21.15.16:0/3668839989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 msgr2=0x55ec09de47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09de47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 -- 172.21.15.16:0/3668839989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 msgr2=0x55ec09def8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09def8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 -- 172.21.15.16:0/3668839989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec09db69b0 con 0x55ec099da980 2024-09-21T02:17:01.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.389+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa584001eb0 tx=0x7fa584005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cea61efcbdf8548 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.390+0000 7fa57bfff640 1 -- 172.21.15.16:0/3668839989 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584012530 con 0x55ec099da980 2024-09-21T02:17:01.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.390+0000 7fa57bfff640 1 -- 172.21.15.16:0/3668839989 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa584049030 con 0x55ec099da980 2024-09-21T02:17:01.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.390+0000 7fa57bfff640 1 -- 172.21.15.16:0/3668839989 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584005150 con 0x55ec099da980 2024-09-21T02:17:01.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- 172.21.15.16:0/3668839989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 msgr2=0x55ec09de42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa584001eb0 tx=0x7fa584005fd0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- 172.21.15.16:0/3668839989 shutdown_connections 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec09a03840 0x55ec09def8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099e2b60 0x55ec09de47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3668839989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec099da980 0x55ec09de42a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- 172.21.15.16:0/3668839989 >> 172.21.15.16:0/3668839989 conn(0x55ec099f7990 msgr2=0x55ec09dac130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- 172.21.15.16:0/3668839989 shutdown_connections 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- 172.21.15.16:0/3668839989 wait complete. 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 -- start start 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.391+0000 7fa59fd63740 1 Processor -- start 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 0x55ec09ddf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 0x55ec09ddff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa594078410 con 0x55ec09a03840 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa592ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa592ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:48732/0 (socket says 172.21.15.16:48732) 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa592ffd640 1 -- 172.21.15.16:0/3133514770 learned_addr learned my addr 172.21.15.16:0/3133514770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 0x55ec09ddff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa58bfff640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 0x55ec09ddf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa594077d40 con 0x55ec099e2b60 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941a75d0 con 0x55ec099da980 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.392+0000 7fa59fd63740 10 client.0 _ll_register_callbacks cb 0x7fa5840083f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 -- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 msgr2=0x55ec09ddf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 0x55ec09ddf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 -- 172.21.15.16:0/3133514770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 msgr2=0x55ec09ddff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 0x55ec09ddff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec09d7bc90 con 0x55ec099da980 2024-09-21T02:17:01.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa58bfff640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 0x55ec09ddf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa5937fe640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 0x55ec09ddff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa592ffd640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa584012640 tx=0x7fa584004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=168c6c7e601c70d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa579ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840042f0 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa5840046c0 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa584049030 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa584004490 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa59fd63740 10 client.25798 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.393+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec09e55cf0 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa584049030 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa5840035f0 con 0x55ec099da980 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 10 client.25798 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 10 client.25798 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:01.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 20 client.25798 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec09997570 0x55ec09e584e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 4 client.25798 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x55ec09d1e590 con 0x55ec09997570 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.394+0000 7fa58bfff640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec09997570 0x55ec09e584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.395+0000 7fa579ffb640 10 client.25798 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:01.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.395+0000 7fa58bfff640 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec09997570 0x55ec09e584e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15d2771c81ca5bc6 server_cookie=9f849b7c77e816a8 in_seq=0 out_seq=0 2024-09-21T02:17:01.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa579ffb640 1 -- 172.21.15.16:0/3133514770 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ec09d1e590 con 0x55ec09997570 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa579ffb640 10 client.25798 handle_command_reply: tid=0 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 client.25798 shutdown 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 msgr2=0x55ec09ddf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa584012640 tx=0x7fa584004040 comp rx=0 tx=0).stop 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 shutdown_connections 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55ec09997570 0x55ec09e584e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ec09a03840 0x55ec09ddff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ec099e2b60 0x55ec09ddf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 --2- 172.21.15.16:0/3133514770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55ec099da980 0x55ec09ddf4a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa579ffb640 0 client.25798 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 >> 172.21.15.16:0/3133514770 conn(0x55ec099f7990 msgr2=0x55ec09dabf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 shutdown_connections 2024-09-21T02:17:01.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.666+0000 7fa59fd63740 1 -- 172.21.15.16:0/3133514770 wait complete. 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.667+0000 7fa59fd63740 20 client.25798 trim_cache size 0 max 16384 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 38, 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "ae7f18a5": { 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 38, 2024-09-21T02:17:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:17:01.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.667+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 msgr2=0x7fa5800b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.667+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 0x7fa5800b8d90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa58c01d000 tx=0x7fa58c01a000 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.667+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 msgr2=0x7fa5941a9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.667+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa57c006fd0 tx=0x7fa57c00b8e0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 shutdown_connections 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fa5800b6880 0x7fa5800b8d90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941aec00 0x7fa5941af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa59410ad10 0x7fa5941a9a20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 --2- 172.21.15.16:0/2016716952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa59410a3c0 0x7fa5941a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 >> 172.21.15.16:0/2016716952 conn(0x7fa5940712b0 msgr2=0x7fa59410e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 shutdown_connections 2024-09-21T02:17:01.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:01.668+0000 7fa589ffb640 1 -- 172.21.15.16:0/2016716952 wait complete. 2024-09-21T02:17:01.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:01 smithi123 ceph-mon[33453]: pgmap v656: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 34 MiB/s wr, 662 op/s 2024-09-21T02:17:01.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:01 smithi016 ceph-mon[31245]: pgmap v656: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 34 MiB/s wr, 662 op/s 2024-09-21T02:17:01.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:01 smithi161 ceph-mon[34061]: pgmap v656: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 34 MiB/s wr, 662 op/s 2024-09-21T02:17:03.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:03 smithi123 ceph-mon[33453]: pgmap v657: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 31 MiB/s wr, 620 op/s 2024-09-21T02:17:03.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:03 smithi016 ceph-mon[31245]: pgmap v657: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 31 MiB/s wr, 620 op/s 2024-09-21T02:17:03.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:03 smithi161 ceph-mon[34061]: pgmap v657: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 31 MiB/s wr, 620 op/s 2024-09-21T02:17:04.411 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ae7f18a5', quiesce took 9.5 seconds. Will release after: 15.0 2024-09-21T02:17:05.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:05 smithi123 ceph-mon[33453]: pgmap v658: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 30 MiB/s wr, 605 op/s 2024-09-21T02:17:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:05 smithi016 ceph-mon[31245]: pgmap v658: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 30 MiB/s wr, 605 op/s 2024-09-21T02:17:05.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:05 smithi161 ceph-mon[34061]: pgmap v658: 129 pgs: 129 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 30 MiB/s wr, 605 op/s 2024-09-21T02:17:06.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:06 smithi016 ceph-mon[31245]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T02:17:06.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:06 smithi161 ceph-mon[34061]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T02:17:07.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:06 smithi123 ceph-mon[33453]: osdmap e95: 12 total, 12 up, 12 in 2024-09-21T02:17:07.816 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:07 smithi161 ceph-mon[34061]: pgmap v660: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:17:07.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:07 smithi016 ceph-mon[31245]: pgmap v660: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:17:08.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:07 smithi123 ceph-mon[33453]: pgmap v660: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 21 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:17:09.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:09 smithi016 ceph-mon[31245]: pgmap v661: 129 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 98 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:17:09.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:09 smithi161 ceph-mon[34061]: pgmap v661: 129 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 98 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:17:10.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:09 smithi123 ceph-mon[33453]: pgmap v661: 129 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 98 active+clean; 20 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:17:11.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:11 smithi016 ceph-mon[31245]: pgmap v662: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 245 op/s 2024-09-21T02:17:11.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:11 smithi161 ceph-mon[34061]: pgmap v662: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 245 op/s 2024-09-21T02:17:12.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:11 smithi123 ceph-mon[33453]: pgmap v662: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 245 op/s 2024-09-21T02:17:13.823 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:13 smithi123 ceph-mon[33453]: pgmap v663: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 130 op/s 2024-09-21T02:17:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:13 smithi016 ceph-mon[31245]: pgmap v663: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 130 op/s 2024-09-21T02:17:13.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:13 smithi161 ceph-mon[34061]: pgmap v663: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 130 op/s 2024-09-21T02:17:15.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:15 smithi016 ceph-mon[31245]: pgmap v664: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2024-09-21T02:17:15.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:15 smithi161 ceph-mon[34061]: pgmap v664: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2024-09-21T02:17:16.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:15 smithi123 ceph-mon[33453]: pgmap v664: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.1 MiB/s wr, 27 op/s 2024-09-21T02:17:17.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:16 smithi123 ceph-mon[33453]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T02:17:17.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:16 smithi016 ceph-mon[31245]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T02:17:17.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:16 smithi161 ceph-mon[34061]: osdmap e96: 12 total, 12 up, 12 in 2024-09-21T02:17:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:17 smithi123 ceph-mon[33453]: pgmap v666: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 823 KiB/s wr, 22 op/s 2024-09-21T02:17:18.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:17 smithi016 ceph-mon[31245]: pgmap v666: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 823 KiB/s wr, 22 op/s 2024-09-21T02:17:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:17 smithi161 ceph-mon[34061]: pgmap v666: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 823 KiB/s wr, 22 op/s 2024-09-21T02:17:19.412 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:17:19.691 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:20.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:19 smithi016 ceph-mon[31245]: pgmap v667: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 599 KiB/s wr, 19 op/s 2024-09-21T02:17:20.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:19 smithi161 ceph-mon[34061]: pgmap v667: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 599 KiB/s wr, 19 op/s 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 -- 172.21.15.16:0/2561295128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678110200 msgr2=0x7f4678112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 --2- 172.21.15.16:0/2561295128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678110200 0x7f4678112620 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4670009ca0 tx=0x7f467002a950 comp rx=0 tx=0).stop 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 -- 172.21.15.16:0/2561295128 shutdown_connections 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 --2- 172.21.15.16:0/2561295128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678110200 0x7f4678112620 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 --2- 172.21.15.16:0/2561295128 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102fa0 0x7f467810fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 --2- 172.21.15.16:0/2561295128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678102650 0x7f4678102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 -- 172.21.15.16:0/2561295128 >> 172.21.15.16:0/2561295128 conn(0x7f4678079760 msgr2=0x7f4678079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 -- 172.21.15.16:0/2561295128 shutdown_connections 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.184+0000 7f467f6c6640 1 -- 172.21.15.16:0/2561295128 wait complete. 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 Processor -- start 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 -- start start 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102650 0x7f467819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 0x7f46781a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4678115b50 con 0x7f4678102fa0 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46781159d0 con 0x7f4678110200 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467f6c6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4678115cd0 con 0x7f4678102650 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467cc61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467cc61640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34106/0 (socket says 172.21.15.16:34106) 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467cc61640 1 -- 172.21.15.16:0/1970956685 learned_addr learned my addr 172.21.15.16:0/1970956685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.185+0000 7f467dc63640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 0x7f46781a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:20.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467d462640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102650 0x7f467819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 -- 172.21.15.16:0/1970956685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102650 msgr2=0x7f467819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102650 0x7f467819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 -- 172.21.15.16:0/1970956685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 msgr2=0x7f46781a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 0x7f46781a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 -- 172.21.15.16:0/1970956685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46781a23b0 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467dc63640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 0x7f46781a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467cc61640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f466800d770 tx=0x7f466800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668017070 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46781a26a0 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46680054b0 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668004c60 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.186+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46781a8ec0 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.187+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467806c1f0 con 0x7f4678102fa0 2024-09-21T02:17:20.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.189+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4668004e00 con 0x7f4678102fa0 2024-09-21T02:17:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.190+0000 7f466e7fc640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 0x7f46640b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.190+0000 7f467d462640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 0x7f46640b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.190+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f46680e79e0 con 0x7f4678102fa0 2024-09-21T02:17:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.191+0000 7f467d462640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 0x7f46640b8bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4674005e10 tx=0x7f4674005da0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:20.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.194+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46680adb70 con 0x7f4678102fa0 2024-09-21T02:17:20.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:19 smithi123 ceph-mon[33453]: pgmap v667: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 599 KiB/s wr, 19 op/s 2024-09-21T02:17:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.423+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f467819d630 con 0x7f4678102fa0 2024-09-21T02:17:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.424+0000 7f466e7fc640 1 -- 172.21.15.16:0/1970956685 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f467819d630 con 0x7f4678102fa0 2024-09-21T02:17:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:17:20.425 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:17:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:17:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.426+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 msgr2=0x7f46640b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.426+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 0x7f46640b8bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4674005e10 tx=0x7f4674005da0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 msgr2=0x7f467819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f466800d770 tx=0x7f466800dc30 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 shutdown_connections 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f46640b66b0 0x7f46640b8bc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4678110200 0x7f46781a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4678102fa0 0x7f467819caa0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 --2- 172.21.15.16:0/1970956685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4678102650 0x7f467819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 >> 172.21.15.16:0/1970956685 conn(0x7f4678079760 msgr2=0x7f4678109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 shutdown_connections 2024-09-21T02:17:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:20.427+0000 7f467f6c6640 1 -- 172.21.15.16:0/1970956685 wait complete. 2024-09-21T02:17:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:20 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1970956685' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:21.253 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id ae7f18a5 --release --await' 2024-09-21T02:17:21.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id ae7f18a5 --release --await 2024-09-21T02:17:21.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:20 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1970956685' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:21.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:20 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1970956685' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 -- 172.21.15.16:0/1037885342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241062f0 msgr2=0x7f9024106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1037885342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241062f0 0x7f9024106700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9014009900 tx=0x7f901402a410 comp rx=0 tx=0).stop 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 -- 172.21.15.16:0/1037885342 shutdown_connections 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1037885342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9024106c40 0x7f90241106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1037885342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241062f0 0x7f9024106700 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1037885342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90241020a0 0x7f90241024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 -- 172.21.15.16:0/1037885342 >> 172.21.15.16:0/1037885342 conn(0x7f9024079f90 msgr2=0x7f902407a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 -- 172.21.15.16:0/1037885342 shutdown_connections 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.166+0000 7f902c7e0640 1 -- 172.21.15.16:0/1037885342 wait complete. 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.167+0000 7f902c7e0640 1 Processor -- start 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 -- start start 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90241062f0 0x7f902419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9024106c40 0x7f902419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90241139a0 con 0x7f90241020a0 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9024113820 con 0x7f9024106c40 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902c7e0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9024113b20 con 0x7f90241062f0 2024-09-21T02:17:22.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34130/0 (socket says 172.21.15.16:34130) 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 -- 172.21.15.16:0/1047557418 learned_addr learned my addr 172.21.15.16:0/1047557418 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f9029d7b640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90241062f0 0x7f902419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902ad7d640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9024106c40 0x7f902419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 -- 172.21.15.16:0/1047557418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90241062f0 msgr2=0x7f902419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90241062f0 0x7f902419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 -- 172.21.15.16:0/1047557418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9024106c40 msgr2=0x7f902419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9024106c40 0x7f902419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.168+0000 7f902a57c640 1 -- 172.21.15.16:0/1047557418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90241a0480 con 0x7f90241020a0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f902a57c640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f902000d770 tx=0x7f902000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f901b7fe640 1 -- 172.21.15.16:0/1047557418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020017070 con 0x7f90241020a0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f901b7fe640 1 -- 172.21.15.16:0/1047557418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9020009c30 con 0x7f90241020a0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90241a0770 con 0x7f90241020a0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f901b7fe640 1 -- 172.21.15.16:0/1047557418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020003c50 con 0x7f90241020a0 2024-09-21T02:17:22.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.169+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90241a6ed0 con 0x7f90241020a0 2024-09-21T02:17:22.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f901b7fe640 1 -- 172.21.15.16:0/1047557418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f9020010460 con 0x7f90241020a0 2024-09-21T02:17:22.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 Processor -- start 2024-09-21T02:17:22.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 -- start start 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a67ea510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 0x5604a67eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 0x5604a67eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604a66d5dc0 con 0x5604a6423840 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5604a67621f0 con 0x5604a6444520 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.172+0000 7f9032d54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5604a6751620 con 0x5604a641b660 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f9029d7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 0x5604a67eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f9029d7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 0x5604a67eaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34134/0 (socket says 172.21.15.16:34134) 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a67ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f9029d7b640 1 -- 172.21.15.16:0/4070884803 learned_addr learned my addr 172.21.15.16:0/4070884803 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902a57c640 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 0x5604a67eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 -- 172.21.15.16:0/4070884803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 msgr2=0x5604a67eaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 0x5604a67eaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 -- 172.21.15.16:0/4070884803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 msgr2=0x5604a67eaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 0x5604a67eaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f901b7fe640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 0x7f90100b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 -- 172.21.15.16:0/4070884803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a6719fd0 con 0x5604a6444520 2024-09-21T02:17:22.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902a57c640 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 0x5604a67eaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f901b7fe640 1 -- 172.21.15.16:0/1047557418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f90200e7430 con 0x7f90241020a0 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902a57c640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 0x7f90100b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.173+0000 7f902ad7d640 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a67ea510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f901c009eb0 tx=0x7f901c010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e760408222d8eb0c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.174+0000 7f9018ff9640 1 -- 172.21.15.16:0/4070884803 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c016070 con 0x5604a6444520 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.174+0000 7f9018ff9640 1 -- 172.21.15.16:0/4070884803 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f901c004d20 con 0x5604a6444520 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.174+0000 7f9018ff9640 1 -- 172.21.15.16:0/4070884803 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c0126b0 con 0x5604a6444520 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.174+0000 7f9032d54740 1 -- 172.21.15.16:0/4070884803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 msgr2=0x5604a67ea510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.174+0000 7f902a57c640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 0x7f90100b8b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f90200e4870 tx=0x7f902000df10 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a67ea510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f901c009eb0 tx=0x7f901c010740 comp rx=0 tx=0).stop 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 -- 172.21.15.16:0/4070884803 shutdown_connections 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6423840 0x5604a67eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a641b660 0x5604a67eaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 --2- 172.21.15.16:0/4070884803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a67ea510 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 -- 172.21.15.16:0/4070884803 >> 172.21.15.16:0/4070884803 conn(0x5604a6438910 msgr2=0x5604a67e8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 -- 172.21.15.16:0/4070884803 shutdown_connections 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 -- 172.21.15.16:0/4070884803 wait complete. 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 -- start start 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.175+0000 7f9032d54740 1 Processor -- start 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a6801e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604a66ec300 con 0x5604a641b660 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5604a6755430 con 0x5604a6444520 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5604a65b2bb0 con 0x5604a6423840 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 10 client.0 _ll_register_callbacks cb 0x5604a6706120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:22.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34140/0 (socket says 172.21.15.16:34140) 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 -- 172.21.15.16:0/2362219587 learned_addr learned my addr 172.21.15.16:0/2362219587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9029d7b640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 -- 172.21.15.16:0/2362219587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 msgr2=0x5604a67f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 -- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 msgr2=0x5604a6801e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a6801e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 -- 172.21.15.16:0/2362219587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a66db780 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f902ad7d640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f901c006010 tx=0x7f901c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fc3ef2272258d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f900dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c02a060 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901c004d20 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.176+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f901c0052a0 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f901c016070 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 10 client.16149 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:22.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5604a64afb00 con 0x5604a641b660 2024-09-21T02:17:22.180 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:21 smithi016 ceph-mon[31245]: pgmap v668: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 166 KiB/s wr, 12 op/s 2024-09-21T02:17:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f901c004d20 con 0x5604a641b660 2024-09-21T02:17:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f901c00f040 con 0x5604a641b660 2024-09-21T02:17:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 10 client.16149 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:22.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 10 client.16149 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 20 client.16149 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a6402ea0 0x5604a6807db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 4 client.16149 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.177+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604a6764140 con 0x5604a6402ea0 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.178+0000 7f9029d7b640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a6402ea0 0x5604a6807db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.178+0000 7f900dffb640 10 client.16149 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.178+0000 7f9029d7b640 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a6402ea0 0x5604a6807db0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=785853137c0cc1d3 server_cookie=30ea2b1e77c1c7ca in_seq=0 out_seq=0 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f900dffb640 1 -- 172.21.15.16:0/2362219587 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5604a6764140 con 0x5604a6402ea0 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f900dffb640 10 client.16149 handle_command_reply: tid=0 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 client.16149 shutdown 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 msgr2=0x5604a67f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f901c006010 tx=0x7f901c009bf0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 shutdown_connections 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a6402ea0 0x5604a6807db0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a6444520 0x5604a6801e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 --2- 172.21.15.16:0/2362219587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a641b660 0x5604a67f5bb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f900dffb640 0 client.16149 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 >> 172.21.15.16:0/2362219587 conn(0x5604a6438910 msgr2=0x5604a67ed6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 shutdown_connections 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.179+0000 7f9032d54740 1 -- 172.21.15.16:0/2362219587 wait complete. 2024-09-21T02:17:22.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.180+0000 7f9032d54740 20 client.16149 trim_cache size 0 max 16384 2024-09-21T02:17:22.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 Processor -- start 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 -- start start 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a6801d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90241b0f10 con 0x5604a6444520 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90241b2a50 con 0x5604a641b660 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9032d54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f90241b15e0 con 0x5604a6423840 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34142/0 (socket says 172.21.15.16:34142) 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 -- 172.21.15.16:0/1281878937 learned_addr learned my addr 172.21.15.16:0/1281878937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a6801d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f9029d7b640 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 -- 172.21.15.16:0/1281878937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 msgr2=0x5604a67f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 -- 172.21.15.16:0/1281878937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 msgr2=0x5604a6801d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a6801d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.197+0000 7f902ad7d640 1 -- 172.21.15.16:0/1281878937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a6526960 con 0x5604a6444520 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.198+0000 7f902ad7d640 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f901c005990 tx=0x7f901c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae5628d07befdc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.198+0000 7f900ffff640 1 -- 172.21.15.16:0/1281878937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c02a030 con 0x5604a6444520 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.198+0000 7f900ffff640 1 -- 172.21.15.16:0/1281878937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f901c01a610 con 0x5604a6444520 2024-09-21T02:17:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.198+0000 7f900ffff640 1 -- 172.21.15.16:0/1281878937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c01ab70 con 0x5604a6444520 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- 172.21.15.16:0/1281878937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 msgr2=0x5604a67f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f901c005990 tx=0x7f901c010740 comp rx=0 tx=0).stop 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- 172.21.15.16:0/1281878937 shutdown_connections 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f0a50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- 172.21.15.16:0/1281878937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a6801d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- 172.21.15.16:0/1281878937 >> 172.21.15.16:0/1281878937 conn(0x5604a6438910 msgr2=0x5604a67f0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- 172.21.15.16:0/1281878937 shutdown_connections 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- 172.21.15.16:0/1281878937 wait complete. 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- start start 2024-09-21T02:17:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 Processor -- start 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a67fe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90241b2380 con 0x5604a6444520 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90241b1cb0 con 0x5604a641b660 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.199+0000 7f9032d54740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f90241aef10 con 0x5604a6423840 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f9032d54740 10 client.0 _ll_register_callbacks cb 0x7f9014005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34144/0 (socket says 172.21.15.16:34144) 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 -- 172.21.15.16:0/824725022 learned_addr learned my addr 172.21.15.16:0/824725022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f9029d7b640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a67fe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 -- 172.21.15.16:0/824725022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 msgr2=0x5604a67f98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 -- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 msgr2=0x5604a67fe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a67fe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 -- 172.21.15.16:0/824725022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a67fa400 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f902ad7d640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f901c0098b0 tx=0x7f901c005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6e23a34816be78 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f900dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c02a040 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f901c0054b0 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.200+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901c01a610 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f901c04b050 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 10 client.16155 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5604a68724b0 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f901c01a610 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f901c022030 con 0x5604a6444520 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 10 client.16155 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 10 client.16155 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 20 client.16155 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:22.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a63d8250 0x5604a6874a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 4 client.16155 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ae7f18a5", "release": true, "await": true}] 2024-09-21T02:17:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ae7f18a5", "release": true, "await": true}) -- 0x5604a675fbf0 con 0x5604a63d8250 2024-09-21T02:17:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.201+0000 7f9029d7b640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a63d8250 0x5604a6874a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.202+0000 7f9029d7b640 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a63d8250 0x5604a6874a80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caeee50a6845f52f server_cookie=93f208de1e828d78 in_seq=0 out_seq=0 2024-09-21T02:17:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.202+0000 7f900dffb640 10 client.16155 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f900dffb640 1 -- 172.21.15.16:0/824725022 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5604a675fbf0 con 0x5604a63d8250 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f900dffb640 10 client.16155 handle_command_reply: tid=0 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 client.16155 shutdown 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 msgr2=0x5604a67f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f901c0098b0 tx=0x7f901c005cf0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 shutdown_connections 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x5604a63d8250 0x5604a6874a80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604a6444520 0x5604a67f9e10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5604a6423840 0x5604a67f98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 --2- 172.21.15.16:0/824725022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5604a641b660 0x5604a67fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 >> 172.21.15.16:0/824725022 conn(0x5604a6438910 msgr2=0x5604a67f77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f900dffb640 0 client.16155 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 shutdown_connections 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 1 -- 172.21.15.16:0/824725022 wait complete. 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.207+0000 7f9032d54740 20 client.16155 trim_cache size 0 max 16384 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:17:22.209 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 40, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "ae7f18a5": { 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 40, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:17:22.210 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:22.211 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 msgr2=0x7f90100b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 0x7f90100b8b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f90200e4870 tx=0x7f902000df10 comp rx=0 tx=0).stop 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 msgr2=0x7f902419a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f902000d770 tx=0x7f902000dc30 comp rx=0 tx=0).stop 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 shutdown_connections 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f90100b6680 0x7f90100b8b90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9024106c40 0x7f902419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90241062f0 0x7f902419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.208+0000 7f902c7e0640 1 --2- 172.21.15.16:0/1047557418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90241020a0 0x7f902419a510 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.209+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 >> 172.21.15.16:0/1047557418 conn(0x7f9024079f90 msgr2=0x7f9024105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.209+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 shutdown_connections 2024-09-21T02:17:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:22.209+0000 7f902c7e0640 1 -- 172.21.15.16:0/1047557418 wait complete. 2024-09-21T02:17:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:21 smithi123 ceph-mon[33453]: pgmap v668: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 166 KiB/s wr, 12 op/s 2024-09-21T02:17:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:21 smithi161 ceph-mon[34061]: pgmap v668: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 166 KiB/s wr, 12 op/s 2024-09-21T02:17:22.991 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ae7f18a5', total seconds elapsed: 28.0 2024-09-21T02:17:22.991 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-09-21T02:17:22.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:17:23.627 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:24.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:23 smithi016 ceph-mon[31245]: pgmap v669: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T02:17:24.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:23 smithi123 ceph-mon[33453]: pgmap v669: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T02:17:24.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.344+0000 7f8962d5b640 1 -- 172.21.15.16:0/1336634909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 msgr2=0x7f895c117b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:24.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.344+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1336634909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c117b40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f894c009900 tx=0x7f894c02a440 comp rx=0 tx=0).stop 2024-09-21T02:17:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.348+0000 7f8962d5b640 1 -- 172.21.15.16:0/1336634909 shutdown_connections 2024-09-21T02:17:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.348+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1336634909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c117b40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.348+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1336634909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c112dc0 0x7f895c1151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.348+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1336634909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f895c10bff0 0x7f895c10c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.348+0000 7f8962d5b640 1 -- 172.21.15.16:0/1336634909 >> 172.21.15.16:0/1336634909 conn(0x7f895c0712f0 msgr2=0x7f895c073e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:24.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.359+0000 7f8962d5b640 1 -- 172.21.15.16:0/1336634909 shutdown_connections 2024-09-21T02:17:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.364+0000 7f8962d5b640 1 -- 172.21.15.16:0/1336634909 wait complete. 2024-09-21T02:17:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.365+0000 7f8962d5b640 1 Processor -- start 2024-09-21T02:17:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.365+0000 7f8962d5b640 1 -- start start 2024-09-21T02:17:24.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.365+0000 7f8962d5b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.366+0000 7f8962d5b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f895c112dc0 0x7f895c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.366+0000 7f8962d5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.366+0000 7f8962d5b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f895c119fc0 con 0x7f895c10bff0 2024-09-21T02:17:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.366+0000 7f8962d5b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f895c119e40 con 0x7f895c115720 2024-09-21T02:17:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.366+0000 7f8962d5b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f895c11a140 con 0x7f895c112dc0 2024-09-21T02:17:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.367+0000 7f8960af7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.367+0000 7f8960af7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34162/0 (socket says 172.21.15.16:34162) 2024-09-21T02:17:24.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.367+0000 7f8960af7640 1 -- 172.21.15.16:0/1729278292 learned_addr learned my addr 172.21.15.16:0/1729278292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.367+0000 7f89612f8640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f89612f8640 1 -- 172.21.15.16:0/1729278292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f895c112dc0 msgr2=0x7f895c1a07c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f89612f8640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f895c112dc0 0x7f895c1a07c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f89612f8640 1 -- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 msgr2=0x7f895c1a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f89612f8640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f89612f8640 1 -- 172.21.15.16:0/1729278292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c1a1540 con 0x7f895c115720 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.371+0000 7f8960af7640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.372+0000 7f89612f8640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c1a0d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f894c009850 tx=0x7f894c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.373+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c03f070 con 0x7f895c115720 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.373+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f894c003e10 con 0x7f895c115720 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.373+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c035030 con 0x7f895c115720 2024-09-21T02:17:24.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.373+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c1a6160 con 0x7f895c115720 2024-09-21T02:17:24.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.373+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1a65c0 con 0x7f895c115720 2024-09-21T02:17:24.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.376+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c10a5d0 con 0x7f895c115720 2024-09-21T02:17:24.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.380+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f894c042070 con 0x7f895c115720 2024-09-21T02:17:24.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.381+0000 7f8959ffb640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 0x7f89440b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:24.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.381+0000 7f8960af7640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 0x7f89440b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:24.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.381+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f894c108770 con 0x7f895c115720 2024-09-21T02:17:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.384+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f894c049070 con 0x7f895c115720 2024-09-21T02:17:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.388+0000 7f8960af7640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 0x7f89440b8d10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8950010ae0 tx=0x7f8950012040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:24.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:23 smithi161 ceph-mon[34061]: pgmap v669: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-21T02:17:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:17:24.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:17:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.850+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f895c1a6980 con 0x7f895c115720 2024-09-21T02:17:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.854+0000 7f8959ffb640 1 -- 172.21.15.16:0/1729278292 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f894c0ce980 con 0x7f895c115720 2024-09-21T02:17:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:17:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 msgr2=0x7f89440b8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 0x7f89440b8d10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8950010ae0 tx=0x7f8950012040 comp rx=0 tx=0).stop 2024-09-21T02:17:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 msgr2=0x7f895c1a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c1a0d20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f894c009850 tx=0x7f894c003b90 comp rx=0 tx=0).stop 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 shutdown_connections 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f89440b6800 0x7f89440b8d10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f895c115720 0x7f895c1a0d20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f895c112dc0 0x7f895c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 --2- 172.21.15.16:0/1729278292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c10bff0 0x7f895c1a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 >> 172.21.15.16:0/1729278292 conn(0x7f895c0712f0 msgr2=0x7f895c115b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 shutdown_connections 2024-09-21T02:17:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:24.862+0000 7f8962d5b640 1 -- 172.21.15.16:0/1729278292 wait complete. 2024-09-21T02:17:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:24 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1729278292' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:25.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:24 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1729278292' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:25.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:24 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1729278292' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:26.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:25 smithi016 ceph-mon[31245]: pgmap v670: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 2.0 MiB/s wr, 52 op/s 2024-09-21T02:17:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:25 smithi123 ceph-mon[33453]: pgmap v670: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 2.0 MiB/s wr, 52 op/s 2024-09-21T02:17:26.283 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0' 2024-09-21T02:17:26.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db / --timeout 90 --expiration 144.5 --await-for 45.0 2024-09-21T02:17:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:25 smithi161 ceph-mon[34061]: pgmap v670: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 2.0 MiB/s wr, 52 op/s 2024-09-21T02:17:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:28.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:27 smithi123 ceph-mon[33453]: pgmap v671: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 96 op/s 2024-09-21T02:17:28.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:27 smithi016 ceph-mon[31245]: pgmap v671: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 96 op/s 2024-09-21T02:17:28.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:27 smithi161 ceph-mon[34061]: pgmap v671: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 96 op/s 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c45378640 1 -- 172.21.15.16:0/1240010852 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38031dd0 con 0x7f8c4010ad10 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 -- 172.21.15.16:0/1240010852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 msgr2=0x7f8c4011cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 --2- 172.21.15.16:0/1240010852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c4011cba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009ca0 tx=0x7f8c3802a8d0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 -- 172.21.15.16:0/1240010852 shutdown_connections 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 --2- 172.21.15.16:0/1240010852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4011d0e0 0x7f8c4011f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 --2- 172.21.15.16:0/1240010852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c4011cba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 --2- 172.21.15.16:0/1240010852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4010a3c0 0x7f8c4010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 -- 172.21.15.16:0/1240010852 >> 172.21.15.16:0/1240010852 conn(0x7f8c40070e80 msgr2=0x7f8c40071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 -- 172.21.15.16:0/1240010852 shutdown_connections 2024-09-21T02:17:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.418+0000 7f8c485de640 1 -- 172.21.15.16:0/1240010852 wait complete. 2024-09-21T02:17:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.419+0000 7f8c485de640 1 Processor -- start 2024-09-21T02:17:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.419+0000 7f8c485de640 1 -- start start 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4010a3c0 0x7f8c4011a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4011d0e0 0x7f8c40113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40122800 con 0x7f8c4011d0e0 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40122680 con 0x7f8c4010ad10 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c485de640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40122980 con 0x7f8c4010a3c0 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c45b79640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c45b79640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:50506/0 (socket says 172.21.15.16:50506) 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.420+0000 7f8c45b79640 1 -- 172.21.15.16:0/1121864589 learned_addr learned my addr 172.21.15.16:0/1121864589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 -- 172.21.15.16:0/1121864589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4010a3c0 msgr2=0x7f8c4011a490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4010a3c0 0x7f8c4011a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 -- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4011d0e0 msgr2=0x7f8c40113d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4011d0e0 0x7f8c40113d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 -- 172.21.15.16:0/1121864589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c40114590 con 0x7f8c4010ad10 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c45b79640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c3802adf0 tx=0x7f8c38032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c377fe640 1 -- 172.21.15.16:0/1121864589 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38003bf0 con 0x7f8c4010ad10 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c401191c0 con 0x7f8c4010ad10 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.421+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c401197d0 con 0x7f8c4010ad10 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.422+0000 7f8c377fe640 1 -- 172.21.15.16:0/1121864589 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c38003d90 con 0x7f8c4010ad10 2024-09-21T02:17:29.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.423+0000 7f8c377fe640 1 -- 172.21.15.16:0/1121864589 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38030b60 con 0x7f8c4010ad10 2024-09-21T02:17:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.426+0000 7f8c4eb52740 1 Processor -- start 2024-09-21T02:17:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.427+0000 7f8c4eb52740 1 -- start start 2024-09-21T02:17:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.427+0000 7f8c377fe640 1 -- 172.21.15.16:0/1121864589 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8c38004040 con 0x7f8c4010ad10 2024-09-21T02:17:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.427+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e66980 0x55f2e4236b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4237080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2e41acb90 con 0x55f2e3e66980 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f2e419bfc0 con 0x55f2e3e6eb60 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f2e419fdd0 con 0x55f2e3e8f840 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c377fe640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 0x7f8c2c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e66980 0x55f2e4236b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:43450/0 (socket says 172.21.15.16:43450) 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3604512411 learned_addr learned my addr 172.21.15.16:0/3604512411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 0x7f8c2c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c377fe640 1 -- 172.21.15.16:0/1121864589 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f8c38108d40 con 0x7f8c4010ad10 2024-09-21T02:17:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3604512411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 msgr2=0x55f2e4237080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4237080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3604512411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e66980 msgr2=0x55f2e4236b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e66980 0x55f2e4236b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.428+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3604512411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e4123060 con 0x55f2e3e8f840 2024-09-21T02:17:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.429+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0127b0 tx=0x7f8c3c012c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e62ebc4c73e151dc server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.429+0000 7f8c34ff9640 1 -- 172.21.15.16:0/3604512411 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c020030 con 0x55f2e3e8f840 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.429+0000 7f8c34ff9640 1 -- 172.21.15.16:0/3604512411 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c3c004c30 con 0x55f2e3e8f840 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.429+0000 7f8c34ff9640 1 -- 172.21.15.16:0/3604512411 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c01c400 con 0x55f2e3e8f840 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3604512411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 msgr2=0x55f2e4236600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0127b0 tx=0x7f8c3c012c70 comp rx=0 tx=0).stop 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 0x7f8c2c0b8c10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c020000 tx=0x7f8c3c01f000 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3604512411 shutdown_connections 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4237080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e66980 0x55f2e4236b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3604512411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e4236600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3604512411 >> 172.21.15.16:0/3604512411 conn(0x55f2e3e83990 msgr2=0x55f2e42344f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3604512411 shutdown_connections 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.430+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3604512411 wait complete. 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 -- start start 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 Processor -- start 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4242720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e424e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2e3ffd7a0 con 0x55f2e3e6eb60 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f2e41410a0 con 0x55f2e3e66980 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f2e4244de0 con 0x55f2e3e8f840 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4eb52740 10 client.0 _ll_register_callbacks cb 0x55f2e411e3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.431+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c45b79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4242720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c45b79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4242720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60086/0 (socket says 172.21.15.16:60086) 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:50532/0 (socket says 172.21.15.16:50532) 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c45b79640 1 -- 172.21.15.16:0/3392950883 learned_addr learned my addr 172.21.15.16:0/3392950883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e424e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 msgr2=0x55f2e42421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3392950883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 msgr2=0x55f2e4242720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4242720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3392950883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e412f880 con 0x55f2e3e8f840 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e424e160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0127b0 tx=0x7f8c3c004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=249b037867961188 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c02b040 con 0x55f2e3e8f840 2024-09-21T02:17:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c004c30 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.432+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f8c3c0054f0 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8c3c01c400 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 10 client.25834 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2e3f09b90 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c3c004c30 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f8c3c01c6b0 con 0x55f2e3e8f840 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 10 client.25834 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 10 client.25834 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 20 client.25834 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e4e1c0 0x55f2e4253fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 4 client.25834 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2e41aeae0 con 0x55f2e3e4e1c0 2024-09-21T02:17:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.433+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e4e1c0 0x55f2e4253fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.434+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e4e1c0 0x55f2e4253fe0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18b1f5a934271a7b server_cookie=d3d407cabb4a8b8 in_seq=0 out_seq=0 2024-09-21T02:17:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.434+0000 7f8c29ffb640 10 client.25834 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.434+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3392950883 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f2e41aeae0 con 0x55f2e3e4e1c0 2024-09-21T02:17:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.434+0000 7f8c29ffb640 10 client.25834 handle_command_reply: tid=0 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 client.25834 shutdown 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 msgr2=0x55f2e424e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e424e160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0127b0 tx=0x7f8c3c004b80 comp rx=0 tx=0).stop 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 shutdown_connections 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e4e1c0 0x55f2e4253fe0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e8f840 0x55f2e424e160 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4242720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c29ffb640 0 client.25834 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3392950883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e66980 0x55f2e42421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 >> 172.21.15.16:0/3392950883 conn(0x55f2e3e83990 msgr2=0x55f2e4239790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 shutdown_connections 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3392950883 wait complete. 2024-09-21T02:17:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.435+0000 7f8c4eb52740 20 client.25834 trim_cache size 0 max 16384 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 Processor -- start 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 -- start start 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4233fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e8f840 0x55f2e423f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2400ffe0 con 0x55f2e3e6eb60 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24011b20 con 0x55f2e3e8f840 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c240106b0 con 0x55f2e3e66980 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:43474/0 (socket says 172.21.15.16:43474) 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 -- 172.21.15.16:0/2402424744 learned_addr learned my addr 172.21.15.16:0/2402424744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c45b79640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4233fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e8f840 0x55f2e423f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 -- 172.21.15.16:0/2402424744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e8f840 msgr2=0x55f2e423f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e8f840 0x55f2e423f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 -- 172.21.15.16:0/2402424744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 msgr2=0x55f2e4233fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4233fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.453+0000 7f8c4637a640 1 -- 172.21.15.16:0/2402424744 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e423d870 con 0x55f2e3e66980 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.454+0000 7f8c45b79640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4233fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.454+0000 7f8c4637a640 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55f2e4235280 tx=0x7f8c300093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=700a7a063f7a12e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.454+0000 7f8c2bfff640 1 -- 172.21.15.16:0/2402424744 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30016a30 con 0x55f2e3e66980 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.454+0000 7f8c2bfff640 1 -- 172.21.15.16:0/2402424744 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c30016bd0 con 0x55f2e3e66980 2024-09-21T02:17:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.454+0000 7f8c2bfff640 1 -- 172.21.15.16:0/2402424744 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3001f3c0 con 0x55f2e3e66980 2024-09-21T02:17:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- 172.21.15.16:0/2402424744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 msgr2=0x55f2e4233aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55f2e4235280 tx=0x7f8c300093e0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- 172.21.15.16:0/2402424744 shutdown_connections 2024-09-21T02:17:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e8f840 0x55f2e423f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e6eb60 0x55f2e4233fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/2402424744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e4233aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- 172.21.15.16:0/2402424744 >> 172.21.15.16:0/2402424744 conn(0x55f2e3e83990 msgr2=0x55f2e423b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- 172.21.15.16:0/2402424744 shutdown_connections 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- 172.21.15.16:0/2402424744 wait complete. 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.455+0000 7f8c4eb52740 1 -- start start 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 Processor -- start 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e424a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4246d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24011450 con 0x55f2e3e8f840 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24010d80 con 0x55f2e3e6eb60 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2400dfe0 con 0x55f2e3e66980 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4eb52740 10 client.0 _ll_register_callbacks cb 0x7f8c3c028040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c46b7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c46b7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60106/0 (socket says 172.21.15.16:60106) 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c46b7b640 1 -- 172.21.15.16:0/3687270085 learned_addr learned my addr 172.21.15.16:0/3687270085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c4637a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e424a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.456+0000 7f8c45b79640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4246d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 -- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 msgr2=0x55f2e4246d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4246d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 -- 172.21.15.16:0/3687270085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 msgr2=0x55f2e4247240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 -- 172.21.15.16:0/3687270085 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e4247980 con 0x55f2e3e66980 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c46b7b640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c45b79640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4246d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4637a640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e424a840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55f2e424e5b0 tx=0x7f8c30021a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d959666413403457 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30021e30 con 0x55f2e3e66980 2024-09-21T02:17:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f8c30028430 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c30016bd0 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8c300059c0 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4eb52740 10 client.25840 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2e42bf860 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.457+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c30016bd0 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f8c30006020 con 0x55f2e3e66980 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 10 client.25840 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 10 client.25840 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 20 client.25840 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e23570 0x55f2e42c2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 4 client.25840 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}] 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 144.5, "await_for": 45.0}) -- 0x55f2e41aa590 con 0x55f2e3e23570 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c45b79640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e23570 0x55f2e42c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c29ffb640 10 client.25840 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.458+0000 7f8c45b79640 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e23570 0x55f2e42c2030 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c96dcb1168c15cb server_cookie=639b95c2e89d8023 in_seq=0 out_seq=0 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c29ffb640 1 -- 172.21.15.16:0/3687270085 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f2e41aa590 con 0x55f2e3e23570 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c29ffb640 10 client.25840 handle_command_reply: tid=0 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 client.25840 shutdown 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 msgr2=0x55f2e424a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e424a840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55f2e424e5b0 tx=0x7f8c30021a30 comp rx=0 tx=0).stop 2024-09-21T02:17:29.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 shutdown_connections 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55f2e3e23570 0x55f2e42c2030 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2e3e8f840 0x55f2e4247240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f2e3e6eb60 0x55f2e4246d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 --2- 172.21.15.16:0/3687270085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f2e3e66980 0x55f2e424a840 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c29ffb640 0 client.25840 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 >> 172.21.15.16:0/3687270085 conn(0x55f2e3e83990 msgr2=0x55f2e4237500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 shutdown_connections 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 1 -- 172.21.15.16:0/3687270085 wait complete. 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.520+0000 7f8c4eb52740 20 client.25840 trim_cache size 0 max 16384 2024-09-21T02:17:29.522 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 42, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "1cac1793": { 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 42, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:17:29.523 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "QUIESCED", 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:29.524 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 msgr2=0x7f8c2c0b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 0x7f8c2c0b8c10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c020000 tx=0x7f8c3c01f000 comp rx=0 tx=0).stop 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 msgr2=0x7f8c40113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c3802adf0 tx=0x7f8c38032f80 comp rx=0 tx=0).stop 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 shutdown_connections 2024-09-21T02:17:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c2c0b6700 0x7f8c2c0b8c10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4011d0e0 0x7f8c40113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4010ad10 0x7f8c40113760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 --2- 172.21.15.16:0/1121864589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4010a3c0 0x7f8c4011a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 >> 172.21.15.16:0/1121864589 conn(0x7f8c40070e80 msgr2=0x7f8c40071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 shutdown_connections 2024-09-21T02:17:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:29.521+0000 7f8c485de640 1 -- 172.21.15.16:0/1121864589 wait complete. 2024-09-21T02:17:30.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:30 smithi016 ceph-mon[31245]: pgmap v672: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 135 op/s 2024-09-21T02:17:30.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:30 smithi123 ceph-mon[33453]: pgmap v672: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 135 op/s 2024-09-21T02:17:30.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:30 smithi161 ceph-mon[34061]: pgmap v672: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 135 op/s 2024-09-21T02:17:32.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:32 smithi123 ceph-mon[33453]: pgmap v673: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 251 op/s 2024-09-21T02:17:32.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:32 smithi016 ceph-mon[31245]: pgmap v673: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 251 op/s 2024-09-21T02:17:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:32 smithi161 ceph-mon[34061]: pgmap v673: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 251 op/s 2024-09-21T02:17:33.130 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1cac1793', quiesce took 10.1 seconds. Will release after: 14.4 2024-09-21T02:17:34.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:34 smithi123 ceph-mon[33453]: pgmap v674: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 307 op/s 2024-09-21T02:17:34.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:34 smithi016 ceph-mon[31245]: pgmap v674: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 307 op/s 2024-09-21T02:17:34.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:34 smithi161 ceph-mon[34061]: pgmap v674: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 307 op/s 2024-09-21T02:17:36.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:36 smithi123 ceph-mon[33453]: pgmap v675: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 15 MiB/s wr, 328 op/s 2024-09-21T02:17:36.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:36 smithi016 ceph-mon[31245]: pgmap v675: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 15 MiB/s wr, 328 op/s 2024-09-21T02:17:36.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:36 smithi161 ceph-mon[34061]: pgmap v675: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 15 MiB/s wr, 328 op/s 2024-09-21T02:17:37.316 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:37 smithi161 ceph-mon[34061]: pgmap v676: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 293 op/s 2024-09-21T02:17:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:37 smithi016 ceph-mon[31245]: pgmap v676: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 293 op/s 2024-09-21T02:17:37.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:37 smithi123 ceph-mon[33453]: pgmap v676: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 293 op/s 2024-09-21T02:17:39.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:39 smithi016 ceph-mon[31245]: pgmap v677: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 255 op/s 2024-09-21T02:17:39.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:39 smithi161 ceph-mon[34061]: pgmap v677: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 255 op/s 2024-09-21T02:17:39.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:39 smithi123 ceph-mon[33453]: pgmap v677: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 255 op/s 2024-09-21T02:17:41.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:41 smithi016 ceph-mon[31245]: pgmap v678: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 212 op/s 2024-09-21T02:17:41.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:41 smithi161 ceph-mon[34061]: pgmap v678: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 212 op/s 2024-09-21T02:17:41.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:41 smithi123 ceph-mon[33453]: pgmap v678: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 212 op/s 2024-09-21T02:17:43.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:43 smithi016 ceph-mon[31245]: pgmap v679: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 97 op/s 2024-09-21T02:17:43.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:43 smithi161 ceph-mon[34061]: pgmap v679: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 97 op/s 2024-09-21T02:17:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:43 smithi123 ceph-mon[33453]: pgmap v679: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 97 op/s 2024-09-21T02:17:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:45 smithi016 ceph-mon[31245]: pgmap v680: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T02:17:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:45 smithi161 ceph-mon[34061]: pgmap v680: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T02:17:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:45 smithi123 ceph-mon[33453]: pgmap v680: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T02:17:47.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:17:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:47 smithi016 ceph-mon[31245]: pgmap v681: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 686 KiB/s wr, 20 op/s 2024-09-21T02:17:47.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:47 smithi161 ceph-mon[34061]: pgmap v681: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 686 KiB/s wr, 20 op/s 2024-09-21T02:17:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:47 smithi123 ceph-mon[33453]: pgmap v681: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 686 KiB/s wr, 20 op/s 2024-09-21T02:17:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 -- 172.21.15.16:0/1115112947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890108cf0 msgr2=0x7fb89010f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 --2- 172.21.15.16:0/1115112947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890108cf0 0x7fb89010f520 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb888009ca0 tx=0x7fb88802a8c0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 -- 172.21.15.16:0/1115112947 shutdown_connections 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 --2- 172.21.15.16:0/1115112947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890108cf0 0x7fb89010f520 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 --2- 172.21.15.16:0/1115112947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 0x7fb8901087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 --2- 172.21.15.16:0/1115112947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890102a60 0x7fb890102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 -- 172.21.15.16:0/1115112947 >> 172.21.15.16:0/1115112947 conn(0x7fb890079730 msgr2=0x7fb890079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 -- 172.21.15.16:0/1115112947 shutdown_connections 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.384+0000 7fb896fd5640 1 -- 172.21.15.16:0/1115112947 wait complete. 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 Processor -- start 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 -- start start 2024-09-21T02:17:48.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 0x7fb8900ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890108cf0 0x7fb8901000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb890112860 con 0x7fb890102a60 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901126e0 con 0x7fb8901083a0 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb896fd5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901129e0 con 0x7fb890108cf0 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb894d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb894d71640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49706/0 (socket says 172.21.15.16:49706) 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb894d71640 1 -- 172.21.15.16:0/98214716 learned_addr learned my addr 172.21.15.16:0/98214716 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb895572640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890108cf0 0x7fb8901000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.385+0000 7fb88ffff640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 0x7fb8900ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 -- 172.21.15.16:0/98214716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890108cf0 msgr2=0x7fb8901000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890108cf0 0x7fb8901000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 -- 172.21.15.16:0/98214716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 msgr2=0x7fb8900ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:48.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 0x7fb8900ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 -- 172.21.15.16:0/98214716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8901a8660 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb894d71640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb88000e9f0 tx=0x7fb88000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88000cd80 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8800054b0 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880010430 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8901a8950 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.386+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8901a8e90 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.387+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb890102e70 con 0x7fb890102a60 2024-09-21T02:17:48.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.389+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fb8800046c0 con 0x7fb890102a60 2024-09-21T02:17:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.390+0000 7fb88dffb640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 0x7fb8780b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.390+0000 7fb88ffff640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 0x7fb8780b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.390+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7fb8800e7fe0 con 0x7fb890102a60 2024-09-21T02:17:48.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.391+0000 7fb88ffff640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 0x7fb8780b8c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb890079c50 tx=0x7fb88400b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:48.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.393+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb8800ae170 con 0x7fb890102a60 2024-09-21T02:17:48.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.619+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb890100970 con 0x7fb890102a60 2024-09-21T02:17:48.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.619+0000 7fb88dffb640 1 -- 172.21.15.16:0/98214716 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fb890100970 con 0x7fb890102a60 2024-09-21T02:17:48.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:17:48.621 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:17:48.621 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.621+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 msgr2=0x7fb8780b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.621+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 0x7fb8780b8c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb890079c50 tx=0x7fb88400b390 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 msgr2=0x7fb8900ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb88000e9f0 tx=0x7fb88000edf0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 shutdown_connections 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fb8780b6700 0x7fb8780b8c10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb890108cf0 0x7fb8901000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8901083a0 0x7fb8900ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 --2- 172.21.15.16:0/98214716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb890102a60 0x7fb8900ff670 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 >> 172.21.15.16:0/98214716 conn(0x7fb890079730 msgr2=0x7fb89006b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 shutdown_connections 2024-09-21T02:17:48.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:48.622+0000 7fb896fd5640 1 -- 172.21.15.16:0/98214716 wait complete. 2024-09-21T02:17:49.329 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --set-id 1cac1793 --release --await' 2024-09-21T02:17:49.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --set-id 1cac1793 --release --await 2024-09-21T02:17:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:17:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:49 smithi016 ceph-mon[31245]: pgmap v682: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 558 KiB/s wr, 17 op/s 2024-09-21T02:17:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:49 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/98214716' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:49 smithi161 ceph-mon[34061]: pgmap v682: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 558 KiB/s wr, 17 op/s 2024-09-21T02:17:49.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:49 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/98214716' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:49.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:49 smithi123 ceph-mon[33453]: pgmap v682: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 558 KiB/s wr, 17 op/s 2024-09-21T02:17:49.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:49 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/98214716' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- 172.21.15.16:0/4113665530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4109df0 msgr2=0x7f19e4110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 --2- 172.21.15.16:0/4113665530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4109df0 0x7f19e4110620 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ca0 tx=0x7f19dc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- 172.21.15.16:0/4113665530 shutdown_connections 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 --2- 172.21.15.16:0/4113665530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e4109df0 0x7f19e4110620 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 --2- 172.21.15.16:0/4113665530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e41094a0 0x7f19e41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 --2- 172.21.15.16:0/4113665530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 0x7f19e4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- 172.21.15.16:0/4113665530 >> 172.21.15.16:0/4113665530 conn(0x7f19e407b850 msgr2=0x7f19e407bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- 172.21.15.16:0/4113665530 shutdown_connections 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- 172.21.15.16:0/4113665530 wait complete. 2024-09-21T02:17:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 Processor -- start 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.116+0000 7f19eabfd640 1 -- start start 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 0x7f19e419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4109df0 0x7f19e419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41139b0 con 0x7f19e41094a0 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4113830 con 0x7f19e4109df0 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19eabfd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4113b30 con 0x7f19e4103b60 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49738/0 (socket says 172.21.15.16:49738) 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 -- 172.21.15.16:0/3570320587 learned_addr learned my addr 172.21.15.16:0/3570320587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e919a640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4109df0 0x7f19e419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e8999640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 0x7f19e419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 -- 172.21.15.16:0/3570320587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 msgr2=0x7f19e419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 0x7f19e419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 -- 172.21.15.16:0/3570320587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4109df0 msgr2=0x7f19e419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4109df0 0x7f19e419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 -- 172.21.15.16:0/3570320587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e41a0250 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.117+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c970 tx=0x7f19d000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.118+0000 7f19e1ffb640 1 -- 172.21.15.16:0/3570320587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0007bb0 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.118+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e41a0540 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.118+0000 7f19e1ffb640 1 -- 172.21.15.16:0/3570320587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d0007d50 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.118+0000 7f19e1ffb640 1 -- 172.21.15.16:0/3570320587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0004cc0 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.118+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e41a6d60 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.120+0000 7f19e1ffb640 1 -- 172.21.15.16:0/3570320587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f19d0004e60 con 0x7f19e41094a0 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.120+0000 7f19f1171740 1 Processor -- start 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 -- start start 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f44108520 0x561f444ae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f44399dc0 con 0x561f440df660 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f444261f0 con 0x561f44108520 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19f1171740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f44415620 con 0x561f440e7840 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49742/0 (socket says 172.21.15.16:49742) 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 -- 172.21.15.16:0/3106298901 learned_addr learned my addr 172.21.15.16:0/3106298901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e919a640 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f44108520 0x561f444ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 -- 172.21.15.16:0/3106298901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 msgr2=0x561f444aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 -- 172.21.15.16:0/3106298901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f44108520 msgr2=0x561f444ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f44108520 0x561f444ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e8999640 1 -- 172.21.15.16:0/3106298901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f443ddfd0 con 0x561f440df660 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.121+0000 7f19e1ffb640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 0x7f19d40b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19e8999640 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f19d800ad00 tx=0x7f19d800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf14d858e6bee5eb server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19e919a640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 0x7f19d40b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19ceffd640 1 -- 172.21.15.16:0/3106298901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d801d070 con 0x561f440df660 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19e1ffb640 1 -- 172.21.15.16:0/3570320587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f19d00e75f0 con 0x7f19e41094a0 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19ceffd640 1 -- 172.21.15.16:0/3106298901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d8018ad0 con 0x561f440df660 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.122+0000 7f19ceffd640 1 -- 172.21.15.16:0/3106298901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8010a20 con 0x561f440df660 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- 172.21.15.16:0/3106298901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 msgr2=0x561f444aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19e919a640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 0x7f19d40b8bc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f19dc006010 tx=0x7f19dc005c30 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f19d800ad00 tx=0x7f19d800f790 comp rx=0 tx=0).stop 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- 172.21.15.16:0/3106298901 shutdown_connections 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440df660 0x561f444aea50 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 --2- 172.21.15.16:0/3106298901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f44108520 0x561f444ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- 172.21.15.16:0/3106298901 >> 172.21.15.16:0/3106298901 conn(0x561f440fc910 msgr2=0x561f444ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- 172.21.15.16:0/3106298901 shutdown_connections 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- 172.21.15.16:0/3106298901 wait complete. 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 -- start start 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.123+0000 7f19f1171740 1 Processor -- start 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444b9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444ba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561f443b0300 con 0x561f44108520 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561f44419430 con 0x561f440df660 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561f44276bb0 con 0x561f440e7840 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19f1171740 10 client.0 _ll_register_callbacks cb 0x561f443ca120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49748/0 (socket says 172.21.15.16:49748) 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 -- 172.21.15.16:0/3559167796 learned_addr learned my addr 172.21.15.16:0/3559167796 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e8999640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444b9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e3fff640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444ba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 -- 172.21.15.16:0/3559167796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 msgr2=0x561f444ba0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444ba0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 -- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 msgr2=0x561f444b9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444b9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 -- 172.21.15.16:0/3559167796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4439f780 con 0x561f44108520 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e8999640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444b9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19e919a640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005cc0 tx=0x7f19dc02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d638cb2efc8b1ba server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.124+0000 7f19c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc004250 con 0x561f44108520 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f19dc0043f0 con 0x561f44108520 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d8018ad0 con 0x561f44108520 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19dc0045f0 con 0x561f44108520 2024-09-21T02:17:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 10 client.16191 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f44173b00 con 0x561f44108520 2024-09-21T02:17:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d8018ad0 con 0x561f44108520 2024-09-21T02:17:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f19dc051070 con 0x561f44108520 2024-09-21T02:17:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 10 client.16191 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 10 client.16191 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 20 client.16191 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f440c6ea0 0x561f444cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 4 client.16191 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.125+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f44428140 con 0x561f440c6ea0 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.126+0000 7f19e8999640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f440c6ea0 0x561f444cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.126+0000 7f19c3fff640 10 client.16191 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.126+0000 7f19e8999640 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f440c6ea0 0x561f444cbef0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08d0589b29cbf20 server_cookie=7298ef3fe2a4a64b in_seq=0 out_seq=0 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19c3fff640 1 -- 172.21.15.16:0/3559167796 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561f44428140 con 0x561f440c6ea0 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19c3fff640 10 client.16191 handle_command_reply: tid=0 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 client.16191 shutdown 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 msgr2=0x561f444ba600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005cc0 tx=0x7f19dc02a8c0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 shutdown_connections 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f440c6ea0 0x561f444cbef0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f444ba600 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19c3fff640 0 client.16191 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 --2- 172.21.15.16:0/3559167796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444b9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 >> 172.21.15.16:0/3559167796 conn(0x561f440fc910 msgr2=0x561f444b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 shutdown_connections 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 1 -- 172.21.15.16:0/3559167796 wait complete. 2024-09-21T02:17:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.127+0000 7f19f1171740 20 client.16191 trim_cache size 0 max 16384 2024-09-21T02:17:50.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 Processor -- start 2024-09-21T02:17:50.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 -- start start 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f44108520 0x561f444bb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b0ed0 con 0x561f440e7840 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b2a10 con 0x561f440df660 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19f1171740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b15a0 con 0x561f44108520 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49752/0 (socket says 172.21.15.16:49752) 2024-09-21T02:17:50.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19e3fff640 1 -- 172.21.15.16:0/1930890967 learned_addr learned my addr 172.21.15.16:0/1930890967 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19e919a640 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f44108520 0x561f444bb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.145+0000 7f19e8999640 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 -- 172.21.15.16:0/1930890967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f44108520 msgr2=0x561f444bb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f44108520 0x561f444bb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 -- 172.21.15.16:0/1930890967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 msgr2=0x561f444afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 -- 172.21.15.16:0/1930890967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f441ea960 con 0x561f440e7840 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19e3fff640 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f19d00aa2f0 tx=0x7f19d000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cade4f78e6f1448f server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1930890967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d001d040 con 0x561f440e7840 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1930890967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d00ed030 con 0x561f440e7840 2024-09-21T02:17:50.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.146+0000 7f19ce7fc640 1 -- 172.21.15.16:0/1930890967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d00e7430 con 0x561f440e7840 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- 172.21.15.16:0/1930890967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 msgr2=0x561f444ba490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f19d00aa2f0 tx=0x7f19d000a340 comp rx=0 tx=0).stop 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- 172.21.15.16:0/1930890967 shutdown_connections 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f44108520 0x561f444bb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f440e7840 0x561f444ba490 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 --2- 172.21.15.16:0/1930890967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- 172.21.15.16:0/1930890967 >> 172.21.15.16:0/1930890967 conn(0x561f440fc910 msgr2=0x561f444bb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- 172.21.15.16:0/1930890967 shutdown_connections 2024-09-21T02:17:50.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- 172.21.15.16:0/1930890967 wait complete. 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 -- start start 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.147+0000 7f19f1171740 1 Processor -- start 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444be670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444bec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b2340 con 0x561f44108520 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b1c70 con 0x561f440df660 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41aeed0 con 0x561f440e7840 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19f1171740 10 client.0 _ll_register_callbacks cb 0x7f19dc0093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49762/0 (socket says 172.21.15.16:49762) 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 -- 172.21.15.16:0/1476311375 learned_addr learned my addr 172.21.15.16:0/1476311375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e3fff640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444bec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e8999640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444be670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 -- 172.21.15.16:0/1476311375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 msgr2=0x561f444bec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444bec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 -- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 msgr2=0x561f444be670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444be670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e919a640 1 -- 172.21.15.16:0/1476311375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f44535f20 con 0x561f44108520 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.148+0000 7f19e8999640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444be670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19e919a640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f19dc053490 tx=0x7f19dc009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308bce8d931d978 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc053a30 con 0x561f44108520 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9482+0+0 (secure 0 0 0) 0x7f19dc053bd0 con 0x561f44108520 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d00ed030 con 0x561f44108520 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19dc053dd0 con 0x561f44108520 2024-09-21T02:17:50.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 10 client.16197 fetch_fsmap learned FSMap version 28 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f445371e0 con 0x561f44108520 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f19d00ed030 con 0x561f44108520 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f19dc051050 con 0x561f44108520 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 10 client.16197 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 10 client.16197 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 20 client.16197 populate_metadata read hostname 'smithi016' 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f4409c250 0x561f44539770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 4 client.16197 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1cac1793", "release": true, "await": true}] 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.149+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1cac1793", "release": true, "await": true}) -- 0x561f44423bf0 con 0x561f4409c250 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.150+0000 7f19e8999640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f4409c250 0x561f44539770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.150+0000 7f19e8999640 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f4409c250 0x561f44539770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eedd8b266a4bef2 server_cookie=3384223581b3fc1b in_seq=0 out_seq=0 2024-09-21T02:17:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.150+0000 7f19c3fff640 10 client.16197 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.155+0000 7f19c3fff640 1 -- 172.21.15.16:0/1476311375 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561f44423bf0 con 0x561f4409c250 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.155+0000 7f19c3fff640 10 client.16197 handle_command_reply: tid=0 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.155+0000 7f19f1171740 1 client.16197 shutdown 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 msgr2=0x561f445358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f19dc053490 tx=0x7f19dc009540 comp rx=0 tx=0).stop 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 shutdown_connections 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x561f4409c250 0x561f44539770 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561f44108520 0x561f445358b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561f440e7840 0x561f444bec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 --2- 172.21.15.16:0/1476311375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x561f440df660 0x561f444be670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 >> 172.21.15.16:0/1476311375 conn(0x561f440fc910 msgr2=0x561f444b5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19c3fff640 0 client.16197 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 shutdown_connections 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 1 -- 172.21.15.16:0/1476311375 wait complete. 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.156+0000 7f19f1171740 20 client.16197 trim_cache size 0 max 16384 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:17:50.158 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 44, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": { 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "1cac1793": { 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "version": 44, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "age_ref": 0.0, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "timeout": 90.0, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "expiration": 144.5, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "members": { 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "file:/": { 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "excluded": false, 2024-09-21T02:17:50.159 INFO:teuthology.orchestra.run.smithi016.stdout: "state": { 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RELEASED", 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:17:50.160 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 msgr2=0x7f19d40b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 0x7f19d40b8bc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f19dc006010 tx=0x7f19dc005c30 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 msgr2=0x7f19e419a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f19d000c970 tx=0x7f19d000cd70 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 shutdown_connections 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f19d40b66b0 0x7f19d40b8bc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19e4109df0 0x7f19e419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19e41094a0 0x7f19e419a940 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 --2- 172.21.15.16:0/3570320587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19e4103b60 0x7f19e419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.157+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 >> 172.21.15.16:0/3570320587 conn(0x7f19e407b850 msgr2=0x7f19e4107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:17:50.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.158+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 shutdown_connections 2024-09-21T02:17:50.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:17:50.158+0000 7f19eabfd640 1 -- 172.21.15.16:0/3570320587 wait complete. 2024-09-21T02:17:51.004 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1cac1793', total seconds elapsed: 28.0 2024-09-21T02:17:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:51 smithi123 ceph-mon[33453]: pgmap v683: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 365 KiB/s wr, 17 op/s 2024-09-21T02:17:51.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:51 smithi016 ceph-mon[31245]: pgmap v683: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 365 KiB/s wr, 17 op/s 2024-09-21T02:17:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:51 smithi161 ceph-mon[34061]: pgmap v683: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 365 KiB/s wr, 17 op/s 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:52 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:52 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:52.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:52 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:53 smithi123 ceph-mon[33453]: pgmap v684: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-09-21T02:17:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:17:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:17:53.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:53.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:53 smithi016 ceph-mon[31245]: pgmap v684: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-09-21T02:17:53.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:17:53.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:17:53.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:53 smithi161 ceph-mon[34061]: pgmap v684: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 34 op/s 2024-09-21T02:17:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:17:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:17:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:17:55.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:55 smithi123 ceph-mon[33453]: pgmap v685: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 136 op/s 2024-09-21T02:17:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:55 smithi016 ceph-mon[31245]: pgmap v685: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 136 op/s 2024-09-21T02:17:55.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:55 smithi161 ceph-mon[34061]: pgmap v685: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 136 op/s 2024-09-21T02:17:57.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:57 smithi123 ceph-mon[33453]: pgmap v686: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 183 op/s 2024-09-21T02:17:57.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:57 smithi016 ceph-mon[31245]: pgmap v686: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 183 op/s 2024-09-21T02:17:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:57 smithi161 ceph-mon[34061]: pgmap v686: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 183 op/s 2024-09-21T02:17:59.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:17:59 smithi123 ceph-mon[33453]: pgmap v687: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 268 op/s 2024-09-21T02:17:59.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:17:59 smithi016 ceph-mon[31245]: pgmap v687: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 268 op/s 2024-09-21T02:17:59.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:17:59 smithi161 ceph-mon[34061]: pgmap v687: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 13 MiB/s wr, 268 op/s 2024-09-21T02:18:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:01 smithi123 ceph-mon[33453]: pgmap v688: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:18:02.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:01 smithi016 ceph-mon[31245]: pgmap v688: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:18:02.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:01 smithi161 ceph-mon[34061]: pgmap v688: 129 pgs: 129 active+clean; 18 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 19 MiB/s wr, 399 op/s 2024-09-21T02:18:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:03 smithi123 ceph-mon[33453]: pgmap v689: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 24 MiB/s wr, 502 op/s 2024-09-21T02:18:04.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:03 smithi016 ceph-mon[31245]: pgmap v689: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 24 MiB/s wr, 502 op/s 2024-09-21T02:18:04.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:03 smithi161 ceph-mon[34061]: pgmap v689: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 24 MiB/s wr, 502 op/s 2024-09-21T02:18:06.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:05 smithi123 ceph-mon[33453]: pgmap v690: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 30 MiB/s wr, 630 op/s 2024-09-21T02:18:06.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:05 smithi016 ceph-mon[31245]: pgmap v690: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 30 MiB/s wr, 630 op/s 2024-09-21T02:18:06.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:05 smithi161 ceph-mon[34061]: pgmap v690: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 30 MiB/s wr, 630 op/s 2024-09-21T02:18:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:08 smithi123 ceph-mon[33453]: pgmap v691: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 27 MiB/s wr, 573 op/s 2024-09-21T02:18:08.411 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:08 smithi016 ceph-mon[31245]: pgmap v691: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 27 MiB/s wr, 573 op/s 2024-09-21T02:18:08.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:08 smithi161 ceph-mon[34061]: pgmap v691: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 27 MiB/s wr, 573 op/s 2024-09-21T02:18:10.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:10 smithi016 ceph-mon[31245]: pgmap v692: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 28 MiB/s wr, 604 op/s 2024-09-21T02:18:10.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:10 smithi161 ceph-mon[34061]: pgmap v692: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 28 MiB/s wr, 604 op/s 2024-09-21T02:18:10.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:10 smithi123 ceph-mon[33453]: pgmap v692: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 28 MiB/s wr, 604 op/s 2024-09-21T02:18:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:11 smithi016 ceph-mon[31245]: pgmap v693: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 30 MiB/s wr, 637 op/s 2024-09-21T02:18:11.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:11 smithi016 ceph-mon[31245]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T02:18:11.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:11 smithi161 ceph-mon[34061]: pgmap v693: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 30 MiB/s wr, 637 op/s 2024-09-21T02:18:11.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:11 smithi161 ceph-mon[34061]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T02:18:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:11 smithi123 ceph-mon[33453]: pgmap v693: 129 pgs: 129 active+clean; 19 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 30 MiB/s wr, 637 op/s 2024-09-21T02:18:11.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:11 smithi123 ceph-mon[33453]: osdmap e97: 12 total, 12 up, 12 in 2024-09-21T02:18:13.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:13 smithi161 ceph-mon[34061]: pgmap v695: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 19 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 28 MiB/s wr, 601 op/s 2024-09-21T02:18:13.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:13 smithi123 ceph-mon[33453]: pgmap v695: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 19 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 28 MiB/s wr, 601 op/s 2024-09-21T02:18:13.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:13 smithi016 ceph-mon[31245]: pgmap v695: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 19 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 28 MiB/s wr, 601 op/s 2024-09-21T02:18:15.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:15 smithi161 ceph-mon[34061]: pgmap v696: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:15.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:15 smithi016 ceph-mon[31245]: pgmap v696: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:15.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:15 smithi123 ceph-mon[33453]: pgmap v696: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:17.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:17 smithi016 ceph-mon[31245]: pgmap v697: 129 pgs: 19 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 59 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 26 MiB/s wr, 548 op/s 2024-09-21T02:18:17.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:17 smithi161 ceph-mon[34061]: pgmap v697: 129 pgs: 19 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 59 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 26 MiB/s wr, 548 op/s 2024-09-21T02:18:17.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:17 smithi123 ceph-mon[33453]: pgmap v697: 129 pgs: 19 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 59 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 26 MiB/s wr, 548 op/s 2024-09-21T02:18:19.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:19 smithi123 ceph-mon[33453]: pgmap v698: 129 pgs: 18 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 70 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 24 MiB/s wr, 507 op/s 2024-09-21T02:18:19.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:19 smithi016 ceph-mon[31245]: pgmap v698: 129 pgs: 18 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 70 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 24 MiB/s wr, 507 op/s 2024-09-21T02:18:19.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:19 smithi161 ceph-mon[34061]: pgmap v698: 129 pgs: 18 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 70 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 24 MiB/s wr, 507 op/s 2024-09-21T02:18:21.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:21 smithi123 ceph-mon[33453]: pgmap v699: 129 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 94 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:18:21.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:21 smithi016 ceph-mon[31245]: pgmap v699: 129 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 94 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:18:21.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:21 smithi161 ceph-mon[34061]: pgmap v699: 129 pgs: 14 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 94 active+clean; 18 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 476 op/s 2024-09-21T02:18:23.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:23 smithi123 ceph-mon[33453]: pgmap v700: 129 pgs: 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 107 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 23 MiB/s wr, 497 op/s 2024-09-21T02:18:23.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:23 smithi016 ceph-mon[31245]: pgmap v700: 129 pgs: 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 107 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 23 MiB/s wr, 497 op/s 2024-09-21T02:18:23.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:23 smithi161 ceph-mon[34061]: pgmap v700: 129 pgs: 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 107 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 23 MiB/s wr, 497 op/s 2024-09-21T02:18:25.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:25 smithi123 ceph-mon[33453]: pgmap v701: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 24 MiB/s wr, 510 op/s 2024-09-21T02:18:25.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:25 smithi016 ceph-mon[31245]: pgmap v701: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 24 MiB/s wr, 510 op/s 2024-09-21T02:18:25.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:25 smithi161 ceph-mon[34061]: pgmap v701: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 24 MiB/s wr, 510 op/s 2024-09-21T02:18:26.709 INFO:tasks.workunit.client.0.smithi016.stdout:stopping iogen 2024-09-21T02:18:26.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'stopping iogen' 2024-09-21T02:18:26.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ ./iogen -k 2024-09-21T02:18:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2024-09-21T02:18:26.718 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2024-09-21T02:18:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:27 smithi123 ceph-mon[33453]: pgmap v702: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 412 op/s 2024-09-21T02:18:27.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:27 smithi016 ceph-mon[31245]: pgmap v702: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 412 op/s 2024-09-21T02:18:27.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:27 smithi161 ceph-mon[34061]: pgmap v702: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 412 op/s 2024-09-21T02:18:28.933 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T02:18:28.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T02:18:29.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:29 smithi016 ceph-mon[31245]: pgmap v703: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 22 MiB/s wr, 473 op/s 2024-09-21T02:18:29.726 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-09-21T02:18:29.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T02:18:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:29 smithi123 ceph-mon[33453]: pgmap v703: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 22 MiB/s wr, 473 op/s 2024-09-21T02:18:29.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:29 smithi161 ceph-mon[34061]: pgmap v703: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 22 MiB/s wr, 473 op/s 2024-09-21T02:18:30.075 DEBUG:teuthology.parallel:result is None 2024-09-21T02:18:30.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T02:18:30.116 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T02:18:30.116 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-21T02:18:30.126 INFO:tasks.quiescer:joining Quiescers 2024-09-21T02:18:30.126 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-21T02:18:30.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:18:30.415 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.937+0000 7ff00d0e7640 1 -- 172.21.15.16:0/2641787271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008109e20 msgr2=0x7ff008110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.937+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/2641787271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008109e20 0x7ff008110650 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff000009900 tx=0x7ff00002a410 comp rx=0 tx=0).stop 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 -- 172.21.15.16:0/2641787271 shutdown_connections 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/2641787271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008109e20 0x7ff008110650 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/2641787271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 0x7ff0081098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/2641787271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008103b90 0x7ff008103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 -- 172.21.15.16:0/2641787271 >> 172.21.15.16:0/2641787271 conn(0x7ff0080ff600 msgr2=0x7ff008101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 -- 172.21.15.16:0/2641787271 shutdown_connections 2024-09-21T02:18:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.938+0000 7ff00d0e7640 1 -- 172.21.15.16:0/2641787271 wait complete. 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.939+0000 7ff00d0e7640 1 Processor -- start 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.939+0000 7ff00d0e7640 1 -- start start 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 0x7ff00819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 0x7ff00819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff008113990 con 0x7ff008103b90 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff008113810 con 0x7ff008109e20 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00d0e7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff008113b10 con 0x7ff0081094d0 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 -- 172.21.15.16:0/1734901780 learned_addr learned my addr 172.21.15.16:0/1734901780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00659c640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 0x7ff00819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00759e640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 0x7ff00819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 -- 172.21.15.16:0/1734901780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 msgr2=0x7ff00819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 0x7ff00819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 -- 172.21.15.16:0/1734901780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 msgr2=0x7ff00819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 0x7ff00819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff006d9d640 1 -- 172.21.15.16:0/1734901780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0081a0210 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.940+0000 7ff00759e640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 0x7ff00819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7ff006d9d640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feff800eab0 tx=0x7feff800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff800cd80 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feff8011900 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0081a0500 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8010870 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.941+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0081a6d20 con 0x7ff008103b90 2024-09-21T02:18:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.942+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff008103fa0 con 0x7ff008103b90 2024-09-21T02:18:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.944+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7feff8011470 con 0x7ff008103b90 2024-09-21T02:18:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.945+0000 7fefeffff640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 0x7feff00b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.945+0000 7ff00659c640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 0x7feff00b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.945+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9570+0+0 (secure 0 0 0) 0x7feff80e80e0 con 0x7ff008103b90 2024-09-21T02:18:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.945+0000 7ff00659c640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 0x7feff00b8c40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff008106f30 tx=0x7feffc00a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:30.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:30.949+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feff8016030 con 0x7ff008103b90 2024-09-21T02:18:31.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.341+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff00819b490 con 0x7ff008103b90 2024-09-21T02:18:31.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.347+0000 7fefeffff640 1 -- 172.21.15.16:0/1734901780 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7feff80ae210 con 0x7ff008103b90 2024-09-21T02:18:31.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:18:31.349 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:18:31.349 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:18:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.352+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 msgr2=0x7feff00b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.352+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 0x7feff00b8c40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff008106f30 tx=0x7feffc00a2d0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.352+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 msgr2=0x7ff00819a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:31.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.352+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feff800eab0 tx=0x7feff800eeb0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 shutdown_connections 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7feff00b6730 0x7feff00b8c40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008109e20 0x7ff00819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0081094d0 0x7ff00819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 --2- 172.21.15.16:0/1734901780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff008103b90 0x7ff00819a360 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:31.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.355+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 >> 172.21.15.16:0/1734901780 conn(0x7ff0080ff600 msgr2=0x7ff008101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:31.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.358+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 shutdown_connections 2024-09-21T02:18:31.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:31.361+0000 7ff00d0e7640 1 -- 172.21.15.16:0/1734901780 wait complete. 2024-09-21T02:18:32.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:31 smithi123 ceph-mon[33453]: pgmap v704: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:32.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:31 smithi123 ceph-mon[33453]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T02:18:32.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:31 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1734901780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:18:32.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:31 smithi016 ceph-mon[31245]: pgmap v704: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:32.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:31 smithi016 ceph-mon[31245]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T02:18:32.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:31 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1734901780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:18:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:31 smithi161 ceph-mon[34061]: pgmap v704: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 27 MiB/s wr, 564 op/s 2024-09-21T02:18:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:31 smithi161 ceph-mon[34061]: osdmap e98: 12 total, 12 up, 12 in 2024-09-21T02:18:32.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:31 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1734901780' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:18:32.484 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24710 quiesce db --cancel --all' 2024-09-21T02:18:32.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- timeout 105 ceph tell mds.24710 quiesce db --cancel --all 2024-09-21T02:18:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 -- 172.21.15.16:0/2069568622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c1094b0 msgr2=0x7f670c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 --2- 172.21.15.16:0/2069568622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c1094b0 0x7f670c1098c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009900 tx=0x7f66fc02a410 comp rx=0 tx=0).stop 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 -- 172.21.15.16:0/2069568622 shutdown_connections 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 --2- 172.21.15.16:0/2069568622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 --2- 172.21.15.16:0/2069568622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c1094b0 0x7f670c1098c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 --2- 172.21.15.16:0/2069568622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c103bc0 0x7f670c103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 -- 172.21.15.16:0/2069568622 >> 172.21.15.16:0/2069568622 conn(0x7f670c07b910 msgr2=0x7f670c07bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 -- 172.21.15.16:0/2069568622 shutdown_connections 2024-09-21T02:18:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.618+0000 7f6713ccd640 1 -- 172.21.15.16:0/2069568622 wait complete. 2024-09-21T02:18:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.619+0000 7f6713ccd640 1 Processor -- start 2024-09-21T02:18:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.619+0000 7f6713ccd640 1 -- start start 2024-09-21T02:18:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 0x7f670c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c1094b0 0x7f670c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f670c113970 con 0x7f670c103bc0 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 0x7f670c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1137f0 con 0x7f670c109e00 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6713ccd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f670c113af0 con 0x7f670c1094b0 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 0x7f670c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54160/0 (socket says 172.21.15.16:54160) 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6711a69640 1 -- 172.21.15.16:0/4026505459 learned_addr learned my addr 172.21.15.16:0/4026505459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f6711268640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c1094b0 0x7f670c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.620+0000 7f671226a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.621+0000 7f671226a640 1 -- 172.21.15.16:0/4026505459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c1094b0 msgr2=0x7f670c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.621+0000 7f671226a640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c1094b0 0x7f670c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.621+0000 7f671226a640 1 -- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 msgr2=0x7f670c19a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.621+0000 7f671226a640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 0x7f670c19a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.621+0000 7f671226a640 1 -- 172.21.15.16:0/4026505459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670c1a0310 con 0x7f670c109e00 2024-09-21T02:18:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f671226a640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c19fbe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6704009fd0 tx=0x7f670400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f6702ffd640 1 -- 172.21.15.16:0/4026505459 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670400ce20 con 0x7f670c109e00 2024-09-21T02:18:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f6702ffd640 1 -- 172.21.15.16:0/4026505459 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6704005020 con 0x7f670c109e00 2024-09-21T02:18:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f6702ffd640 1 -- 172.21.15.16:0/4026505459 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704010690 con 0x7f670c109e00 2024-09-21T02:18:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670c1a0600 con 0x7f670c109e00 2024-09-21T02:18:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.622+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f670c1a6d60 con 0x7f670c109e00 2024-09-21T02:18:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.626+0000 7f671a241740 1 Processor -- start 2024-09-21T02:18:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.627+0000 7f671a241740 1 -- start start 2024-09-21T02:18:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826782d0 0x55c582a1f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f6702ffd640 1 -- 172.21.15.16:0/4026505459 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6704004a40 con 0x7f670c109e00 2024-09-21T02:18:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c582995dd0 con 0x55c58264f410 2024-09-21T02:18:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c582985200 con 0x55c5826782d0 2024-09-21T02:18:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.628+0000 7f671a241740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c582989010 con 0x55c5826575f0 2024-09-21T02:18:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.629+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.629+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54162/0 (socket says 172.21.15.16:54162) 2024-09-21T02:18:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.629+0000 7f6711a69640 1 -- 172.21.15.16:0/662114857 learned_addr learned my addr 172.21.15.16:0/662114857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.630+0000 7f6702ffd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 0x7f66f80b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f671226a640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 0x7f66f80b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f6711a69640 1 -- 172.21.15.16:0/662114857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 msgr2=0x55c582a1fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f6711a69640 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1fd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f6711a69640 1 -- 172.21.15.16:0/662114857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826782d0 msgr2=0x55c582a1f310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f6711a69640 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826782d0 0x55c582a1f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.631+0000 7f6711a69640 1 -- 172.21.15.16:0/662114857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5826e38b0 con 0x55c58264f410 2024-09-21T02:18:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.632+0000 7f6711a69640 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dc0 tx=0x7f6708011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60c3fb5a427b703 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.632+0000 7f6702ffd640 1 -- 172.21.15.16:0/4026505459 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f67040e8180 con 0x7f670c109e00 2024-09-21T02:18:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.632+0000 7f66f7fff640 1 -- 172.21.15.16:0/662114857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708005220 con 0x55c58264f410 2024-09-21T02:18:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.632+0000 7f66f7fff640 1 -- 172.21.15.16:0/662114857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6708004c30 con 0x55c58264f410 2024-09-21T02:18:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.632+0000 7f66f7fff640 1 -- 172.21.15.16:0/662114857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708019810 con 0x55c58264f410 2024-09-21T02:18:33.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.634+0000 7f671a241740 1 -- 172.21.15.16:0/662114857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 msgr2=0x55c582a1f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.634+0000 7f671a241740 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dc0 tx=0x7f6708011810 comp rx=0 tx=0).stop 2024-09-21T02:18:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.636+0000 7f671a241740 1 -- 172.21.15.16:0/662114857 shutdown_connections 2024-09-21T02:18:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.636+0000 7f671a241740 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.636+0000 7f671a241740 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a1f850 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.636+0000 7f671a241740 1 --2- 172.21.15.16:0/662114857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826782d0 0x55c582a1f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.636+0000 7f671a241740 1 -- 172.21.15.16:0/662114857 >> 172.21.15.16:0/662114857 conn(0x55c58266c6c0 msgr2=0x55c582a1d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.640+0000 7f671a241740 1 -- 172.21.15.16:0/662114857 shutdown_connections 2024-09-21T02:18:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.644+0000 7f671226a640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 0x7f66f80b8c90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f67040aa180 tx=0x7f67040aa200 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.645+0000 7f671a241740 1 -- 172.21.15.16:0/662114857 wait complete. 2024-09-21T02:18:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.645+0000 7f671a241740 1 -- start start 2024-09-21T02:18:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.645+0000 7f671a241740 1 Processor -- start 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a2aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 0x55c582a2b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 0x55c582a395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c5827e6810 con 0x55c58264f410 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c582920080 con 0x55c5826575f0 2024-09-21T02:18:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55c582a2daf0 con 0x55c5826782d0 2024-09-21T02:18:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.649+0000 7f671a241740 10 client.0 _ll_register_callbacks cb 0x55c582930960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:18:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.650+0000 7f6711a69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 0x55c582a2b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.657+0000 7f6711a69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 0x55c582a2b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:46056/0 (socket says 172.21.15.16:46056) 2024-09-21T02:18:33.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.657+0000 7f6711a69640 1 -- 172.21.15.16:0/2882766397 learned_addr learned my addr 172.21.15.16:0/2882766397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.659+0000 7f6711268640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 0x55c582a395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.659+0000 7f6711268640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a2aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.671+0000 7f6711268640 1 -- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 msgr2=0x55c582a2b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.671+0000 7f6711268640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 0x55c582a2b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.671+0000 7f6711268640 1 -- 172.21.15.16:0/2882766397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 msgr2=0x55c582a2aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.671+0000 7f6711268640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a2aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.671+0000 7f6711268640 1 -- 172.21.15.16:0/2882766397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c58292a460 con 0x55c5826782d0 2024-09-21T02:18:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.672+0000 7f6711268640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 0x55c582a395f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66fc02a930 tx=0x7f66fc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4742c3f1c3bcd80 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.672+0000 7f66f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-21T02:18:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.672+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc035040 con 0x55c5826782d0 2024-09-21T02:18:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.672+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6708004c30 con 0x55c5826782d0 2024-09-21T02:18:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.672+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f66fc031c70 con 0x55c5826782d0 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f66fc04e050 con 0x55c5826782d0 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 10 client.25873 fetch_fsmap learned FSMap version 28 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c582a53460 con 0x55c5826782d0 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6708004c30 con 0x55c5826782d0 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f66fc003880 con 0x55c5826782d0 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 10 client.25873 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:18:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 10 client.25873 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 20 client.25873 populate_metadata read hostname 'smithi016' 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.673+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582636c50 0x55c582a56d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.674+0000 7f671a241740 4 client.25873 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.674+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c582997d20 con 0x55c582636c50 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.674+0000 7f6711a69640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582636c50 0x55c582a56d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.674+0000 7f66f57fa640 10 client.25873 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:18:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.675+0000 7f6711a69640 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582636c50 0x55c582a56d80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb13bf0dae7c3122 server_cookie=3f6f09c8a81b7378 in_seq=0 out_seq=0 2024-09-21T02:18:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.679+0000 7f66f57fa640 1 -- 172.21.15.16:0/2882766397 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c582997d20 con 0x55c582636c50 2024-09-21T02:18:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.679+0000 7f66f57fa640 10 client.25873 handle_command_reply: tid=0 2024-09-21T02:18:33.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.679+0000 7f671a241740 1 client.25873 shutdown 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 msgr2=0x55c582a395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 0x55c582a395f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66fc02a930 tx=0x7f66fc000bc0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 shutdown_connections 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582636c50 0x55c582a56d80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826782d0 0x55c582a395f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66fc02a930 tx=0x7f66fc000bc0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c5826575f0 0x55c582a2b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 --2- 172.21.15.16:0/2882766397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a2aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 >> 172.21.15.16:0/2882766397 conn(0x55c58266c6c0 msgr2=0x55c582a224a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.684+0000 7f66f57fa640 0 client.25873 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.685+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 shutdown_connections 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.685+0000 7f671a241740 1 -- 172.21.15.16:0/2882766397 wait complete. 2024-09-21T02:18:33.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.685+0000 7f671a241740 20 client.25873 trim_cache size 0 max 16384 2024-09-21T02:18:33.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.721+0000 7f671a241740 1 Processor -- start 2024-09-21T02:18:33.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.722+0000 7f671a241740 1 -- start start 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58264f410 0x55c582a395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58260c000 0x55c582a243a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f670c076e00 con 0x55c58260c000 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a8070 con 0x55c58264f410 2024-09-21T02:18:33.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.723+0000 7f671a241740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f670c0774d0 con 0x55c5826575f0 2024-09-21T02:18:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:46948/0 (socket says 172.21.15.16:46948) 2024-09-21T02:18:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 -- 172.21.15.16:0/4094266752 learned_addr learned my addr 172.21.15.16:0/4094266752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711268640 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58264f410 0x55c582a395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711268640 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58260c000 0x55c582a243a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 -- 172.21.15.16:0/4094266752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58264f410 msgr2=0x55c582a395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58264f410 0x55c582a395e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 -- 172.21.15.16:0/4094266752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58260c000 msgr2=0x55c582a243a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58260c000 0x55c582a243a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.725+0000 7f6711a69640 1 -- 172.21.15.16:0/4094266752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c582a2b360 con 0x55c5826575f0 2024-09-21T02:18:33.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.729+0000 7f6711a69640 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6708014040 tx=0x7f6708011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9384f086ab84bd3d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.731+0000 7f66f77fe640 1 -- 172.21.15.16:0/4094266752 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708031070 con 0x55c5826575f0 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.731+0000 7f66f77fe640 1 -- 172.21.15.16:0/4094266752 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67080045a0 con 0x55c5826575f0 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.732+0000 7f66f77fe640 1 -- 172.21.15.16:0/4094266752 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708019700 con 0x55c5826575f0 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.732+0000 7f671a241740 1 -- 172.21.15.16:0/4094266752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 msgr2=0x55c582a1d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.732+0000 7f671a241740 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6708014040 tx=0x7f6708011810 comp rx=0 tx=0).stop 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.733+0000 7f671a241740 1 -- 172.21.15.16:0/4094266752 shutdown_connections 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.733+0000 7f671a241740 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58260c000 0x55c582a243a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.733+0000 7f671a241740 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a1d270 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.733+0000 7f671a241740 1 --2- 172.21.15.16:0/4094266752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58264f410 0x55c582a395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.733+0000 7f671a241740 1 -- 172.21.15.16:0/4094266752 >> 172.21.15.16:0/4094266752 conn(0x55c58266c6c0 msgr2=0x55c582a1cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.737+0000 7f671a241740 1 -- 172.21.15.16:0/4094266752 shutdown_connections 2024-09-21T02:18:33.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.737+0000 7f671a241740 1 -- 172.21.15.16:0/4094266752 wait complete. 2024-09-21T02:18:33.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.737+0000 7f671a241740 1 -- start start 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 Processor -- start 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58260c000 0x55c582a3d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a3b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a79a0 con 0x55c58264f410 2024-09-21T02:18:33.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a72d0 con 0x55c58260c000 2024-09-21T02:18:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f670c074e00 con 0x55c5826575f0 2024-09-21T02:18:33.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.738+0000 7f671a241740 10 client.0 _ll_register_callbacks cb 0x7f66fc032ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T02:18:33.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.743+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.743+0000 7f6711a69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54186/0 (socket says 172.21.15.16:54186) 2024-09-21T02:18:33.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.743+0000 7f6711a69640 1 -- 172.21.15.16:0/2689786901 learned_addr learned my addr 172.21.15.16:0/2689786901 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:33.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.746+0000 7f6711268640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58260c000 0x55c582a3d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711268640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a3b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711a69640 1 -- 172.21.15.16:0/2689786901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 msgr2=0x55c582a3b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711a69640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a3b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711a69640 1 -- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58260c000 msgr2=0x55c582a3d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711a69640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58260c000 0x55c582a3d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.747+0000 7f6711a69640 1 -- 172.21.15.16:0/2689786901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c582a3b6f0 con 0x55c58264f410 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.748+0000 7f6711a69640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6708009b50 tx=0x7f6708009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03a6ac8158a9be4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.749+0000 7f66f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.749+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708031040 con 0x55c58264f410 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.749+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f67080035f0 con 0x55c58264f410 2024-09-21T02:18:33.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.749+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67080045a0 con 0x55c58264f410 2024-09-21T02:18:33.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.753+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67080037f0 con 0x55c58264f410 2024-09-21T02:18:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.757+0000 7f671a241740 10 client.16221 fetch_fsmap learned FSMap version 28 2024-09-21T02:18:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.757+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c582ac1f20 con 0x55c58264f410 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67080045a0 con 0x55c58264f410 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f670803d030 con 0x55c58264f410 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 10 client.16221 fetch_fsmap finished waiting for FSMap version 28 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 10 client.16221 resolve_mds: validated gid 24710 aka daemon mds.b 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 20 client.16221 populate_metadata read hostname 'smithi016' 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582ac2350 0x55c582ac4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 4 client.16221 mds_command: new command op to 24710 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-21T02:18:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.760+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 --> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55c5829937d0 con 0x55c582ac2350 2024-09-21T02:18:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.761+0000 7f6711268640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582ac2350 0x55c582ac4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:33.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.766+0000 7f6711268640 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582ac2350 0x55c582ac4770 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53992db60cacb4f5 server_cookie=ffd7ff32fc0ab64f in_seq=0 out_seq=0 2024-09-21T02:18:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.767+0000 7f66f57fa640 10 client.16221 ms_handle_connect on v2:172.21.15.123:6838/1116723582 2024-09-21T02:18:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.767+0000 7f66f57fa640 1 -- 172.21.15.16:0/2689786901 <== mds.0 v2:172.21.15.123:6838/1116723582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55c5829937d0 con 0x55c582ac2350 2024-09-21T02:18:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.767+0000 7f66f57fa640 10 client.16221 handle_command_reply: tid=0 2024-09-21T02:18:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.767+0000 7f671a241740 1 client.16221 shutdown 2024-09-21T02:18:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.769+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 msgr2=0x55c582a3d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.769+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6708009b50 tx=0x7f6708009d50 comp rx=0 tx=0).stop 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 shutdown_connections 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:6838/1116723582,v1:172.21.15.123:6839/1116723582] conn(0x55c582ac2350 0x55c582ac4770 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55c5826575f0 0x55c582a3b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c58264f410 0x55c582a3d970 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 --2- 172.21.15.16:0/2689786901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c58260c000 0x55c582a3d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 >> 172.21.15.16:0/2689786901 conn(0x55c58266c6c0 msgr2=0x55c582a2c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.772+0000 7f66f57fa640 0 client.16221 ms_handle_reset on v2:172.21.15.123:6838/1116723582 2024-09-21T02:18:33.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.775+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 shutdown_connections 2024-09-21T02:18:33.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.777+0000 7f671a241740 1 -- 172.21.15.16:0/2689786901 wait complete. 2024-09-21T02:18:33.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.779+0000 7f671a241740 20 client.16221 trim_cache size 0 max 16384 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 28, 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "leader": 24710, 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "set_version": 44, 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "sets": {} 2024-09-21T02:18:33.780 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 msgr2=0x7f66f80b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 0x7f66f80b8c90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f67040aa180 tx=0x7f67040aa200 comp rx=0 tx=0).stop 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 msgr2=0x7f670c19fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c19fbe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6704009fd0 tx=0x7f670400ef10 comp rx=0 tx=0).stop 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 shutdown_connections 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f66f80b6780 0x7f66f80b8c90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f670c109e00 0x7f670c19fbe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f670c1094b0 0x7f670c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 --2- 172.21.15.16:0/4026505459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f670c103bc0 0x7f670c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 >> 172.21.15.16:0/4026505459 conn(0x7f670c07b910 msgr2=0x7f670c105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 shutdown_connections 2024-09-21T02:18:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:33.780+0000 7f6713ccd640 1 -- 172.21.15.16:0/4026505459 wait complete. 2024-09-21T02:18:34.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:33 smithi123 ceph-mon[33453]: pgmap v706: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 26 MiB/s wr, 544 op/s 2024-09-21T02:18:34.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:33 smithi016 ceph-mon[31245]: pgmap v706: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 26 MiB/s wr, 544 op/s 2024-09-21T02:18:34.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:33 smithi161 ceph-mon[34061]: pgmap v706: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 26 MiB/s wr, 544 op/s 2024-09-21T02:18:34.681 INFO:tasks.quiescer:done joining Quiescers 2024-09-21T02:18:34.681 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-21T02:18:34.691 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-21T02:18:34.691 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T02:18:34.692 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:18:34.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-21T02:18:34.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-21T02:18:34.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-21T02:18:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.146+0000 7f736e969640 1 -- 172.21.15.16:0/495392361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368107c10 msgr2=0x7f7368108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.146+0000 7f736e969640 1 --2- 172.21.15.16:0/495392361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368107c10 0x7f7368108080 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f736001acb0 tx=0x7f736003b900 comp rx=0 tx=0).stop 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.150+0000 7f736e969640 1 -- 172.21.15.16:0/495392361 shutdown_connections 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.150+0000 7f736e969640 1 --2- 172.21.15.16:0/495392361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368107c10 0x7f7368108080 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.150+0000 7f736e969640 1 --2- 172.21.15.16:0/495392361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73681030a0 0x7f73681075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.150+0000 7f736e969640 1 --2- 172.21.15.16:0/495392361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7368102750 0x7f7368102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.150+0000 7f736e969640 1 -- 172.21.15.16:0/495392361 >> 172.21.15.16:0/495392361 conn(0x7f73680fe410 msgr2=0x7f7368100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:35.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.151+0000 7f736e969640 1 -- 172.21.15.16:0/495392361 shutdown_connections 2024-09-21T02:18:35.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.151+0000 7f736e969640 1 -- 172.21.15.16:0/495392361 wait complete. 2024-09-21T02:18:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.152+0000 7f736e969640 1 Processor -- start 2024-09-21T02:18:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.152+0000 7f736e969640 1 -- start start 2024-09-21T02:18:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73681030a0 0x7f7368074970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7368107c10 0x7f736810ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f736810b2c0 con 0x7f7368102750 2024-09-21T02:18:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f736810b140 con 0x7f73681030a0 2024-09-21T02:18:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.153+0000 7f736e969640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f736810b440 con 0x7f7368107c10 2024-09-21T02:18:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.155+0000 7f7367fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.155+0000 7f7367fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54204/0 (socket says 172.21.15.16:54204) 2024-09-21T02:18:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.155+0000 7f7367fff640 1 -- 172.21.15.16:0/1649990299 learned_addr learned my addr 172.21.15.16:0/1649990299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.157+0000 7f73677fe640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73681030a0 0x7f7368074970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.157+0000 7f736cf06640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7368107c10 0x7f736810ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.158+0000 7f7367fff640 1 -- 172.21.15.16:0/1649990299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7368107c10 msgr2=0x7f736810ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.158+0000 7f7367fff640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7368107c10 0x7f736810ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.158+0000 7f7367fff640 1 -- 172.21.15.16:0/1649990299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73681030a0 msgr2=0x7f7368074970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.158+0000 7f7367fff640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73681030a0 0x7f7368074970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.158+0000 7f7367fff640 1 -- 172.21.15.16:0/1649990299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736810f490 con 0x7f7368102750 2024-09-21T02:18:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.161+0000 7f7367fff640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f735800ea20 tx=0x7f735800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:35.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.163+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735800cd80 con 0x7f7368102750 2024-09-21T02:18:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.163+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73580108c0 con 0x7f7368102750 2024-09-21T02:18:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.163+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358018710 con 0x7f7368102750 2024-09-21T02:18:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.163+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736810f780 con 0x7f7368102750 2024-09-21T02:18:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.164+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736810fd00 con 0x7f7368102750 2024-09-21T02:18:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.166+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73681075c0 con 0x7f7368102750 2024-09-21T02:18:35.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.169+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7358010430 con 0x7f7368102750 2024-09-21T02:18:35.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.171+0000 7f73657fa640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 0x7f73500b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.171+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f73580e7f20 con 0x7f7368102750 2024-09-21T02:18:35.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.171+0000 7f73677fe640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 0x7f73500b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.173+0000 7f73677fe640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 0x7f73500b8d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f735c006fd0 tx=0x7f735c008040 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:35.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.182+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f73580ae050 con 0x7f7368102750 2024-09-21T02:18:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.536+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7368106cb0 con 0x7f73500b6880 2024-09-21T02:18:35.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.596+0000 7f73657fa640 1 -- 172.21.15.16:0/1649990299 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f7368106cb0 con 0x7f73500b6880 2024-09-21T02:18:35.597 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-21T00:00:00", "created": "2024-09-21T02:07:08", "first": "2024-09-21T02:08:00", "last": "2024-09-21T02:18:01", "last_pruned": "2024-09-21T02:18:01", "created_count": 11, "pruned_count": 5, "active": true} 2024-09-21T02:18:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 msgr2=0x7f73500b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 0x7f73500b8d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f735c006fd0 tx=0x7f735c008040 comp rx=0 tx=0).stop 2024-09-21T02:18:35.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 msgr2=0x7f73680743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f735800ea20 tx=0x7f735800eee0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 shutdown_connections 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f73500b6880 0x7f73500b8d90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7368107c10 0x7f736810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73681030a0 0x7f7368074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 --2- 172.21.15.16:0/1649990299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7368102750 0x7f73680743d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 >> 172.21.15.16:0/1649990299 conn(0x7f73680fe410 msgr2=0x7f7368105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 shutdown_connections 2024-09-21T02:18:35.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.599+0000 7f736e969640 1 -- 172.21.15.16:0/1649990299 wait complete. 2024-09-21T02:18:35.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-21T02:18:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.819+0000 7fdaa3fff640 1 -- 172.21.15.16:0/2016557986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c096f70 msgr2=0x7fda9c097380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.819+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/2016557986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c096f70 0x7fda9c097380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda94009900 tx=0x7fda9402a440 comp rx=0 tx=0).stop 2024-09-21T02:18:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 -- 172.21.15.16:0/2016557986 shutdown_connections 2024-09-21T02:18:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/2016557986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c0a0520 0x7fda9c0a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/2016557986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0979a0 0x7fda9c09ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/2016557986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c096f70 0x7fda9c097380 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 -- 172.21.15.16:0/2016557986 >> 172.21.15.16:0/2016557986 conn(0x7fda9c00ce30 msgr2=0x7fda9c00d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.823+0000 7fdaa3fff640 1 -- 172.21.15.16:0/2016557986 shutdown_connections 2024-09-21T02:18:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.824+0000 7fdaa3fff640 1 -- 172.21.15.16:0/2016557986 wait complete. 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 Processor -- start 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 -- start start 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 0x7fda9c094e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0a0520 0x7fda9c0953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c0a6090 con 0x7fda9c096f70 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c0a5f10 con 0x7fda9c0a0520 2024-09-21T02:18:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.825+0000 7fdaa3fff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c0a6210 con 0x7fda9c0979a0 2024-09-21T02:18:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.830+0000 7fdaa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.830+0000 7fdaa2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54214/0 (socket says 172.21.15.16:54214) 2024-09-21T02:18:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.830+0000 7fdaa2ffd640 1 -- 172.21.15.16:0/387500725 learned_addr learned my addr 172.21.15.16:0/387500725 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.830+0000 7fdaa37fe640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0a0520 0x7fda9c0953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.830+0000 7fdaa27fc640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 0x7fda9c094e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.832+0000 7fdaa2ffd640 1 -- 172.21.15.16:0/387500725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 msgr2=0x7fda9c094e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.832+0000 7fdaa2ffd640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 0x7fda9c094e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.832+0000 7fdaa2ffd640 1 -- 172.21.15.16:0/387500725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0a0520 msgr2=0x7fda9c0953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.832+0000 7fdaa2ffd640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0a0520 0x7fda9c0953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.832+0000 7fdaa2ffd640 1 -- 172.21.15.16:0/387500725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c095be0 con 0x7fda9c096f70 2024-09-21T02:18:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.833+0000 7fdaa27fc640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 0x7fda9c094e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T02:18:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.834+0000 7fdaa2ffd640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fda9402a9b0 tx=0x7fda94003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.836+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9403f070 con 0x7fda9c096f70 2024-09-21T02:18:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.836+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fda9403baf0 con 0x7fda9c096f70 2024-09-21T02:18:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.836+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94035030 con 0x7fda9c096f70 2024-09-21T02:18:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.838+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c141c10 con 0x7fda9c096f70 2024-09-21T02:18:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.838+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c142130 con 0x7fda9c096f70 2024-09-21T02:18:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.842+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c006950 con 0x7fda9c096f70 2024-09-21T02:18:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.845+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fda94042070 con 0x7fda9c096f70 2024-09-21T02:18:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.846+0000 7fda8bfff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 0x7fda8c0b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.847+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7fda94108810 con 0x7fda9c096f70 2024-09-21T02:18:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.847+0000 7fdaa27fc640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 0x7fda8c0b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.848+0000 7fdaa27fc640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 0x7fda8c0b8d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a1d90 tx=0x7fda98009110 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:35.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:35.854+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fda94049030 con 0x7fda9c096f70 2024-09-21T02:18:36.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.263+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 --> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fda9c099f20 con 0x7fda8c0b6880 2024-09-21T02:18:36.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:36 smithi123 ceph-mon[33453]: pgmap v707: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 18 MiB/s wr, 457 op/s 2024-09-21T02:18:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.346+0000 7fda8bfff640 1 -- 172.21.15.16:0/387500725 <== mgr.14223 v2:172.21.15.16:6800/3647438626 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fda9c099f20 con 0x7fda8c0b6880 2024-09-21T02:18:36.347 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 msgr2=0x7fda8c0b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 0x7fda8c0b8d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a1d90 tx=0x7fda98009110 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 msgr2=0x7fda9c0968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fda9402a9b0 tx=0x7fda94003b90 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 shutdown_connections 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fda8c0b6880 0x7fda8c0b8d90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda9c0a0520 0x7fda9c0953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda9c0979a0 0x7fda9c094e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 --2- 172.21.15.16:0/387500725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c096f70 0x7fda9c0968a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 >> 172.21.15.16:0/387500725 conn(0x7fda9c00ce30 msgr2=0x7fda9c092e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 shutdown_connections 2024-09-21T02:18:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.349+0000 7fdaa3fff640 1 -- 172.21.15.16:0/387500725 wait complete. 2024-09-21T02:18:36.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:36 smithi016 ceph-mon[31245]: pgmap v707: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 18 MiB/s wr, 457 op/s 2024-09-21T02:18:36.369 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-21T02:18:36.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T02:18:36.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:36 smithi161 ceph-mon[34061]: pgmap v707: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 18 MiB/s wr, 457 op/s 2024-09-21T02:18:36.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.569+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1108079191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea4102980 msgr2=0x7f7ea4102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.569+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1108079191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea4102980 0x7f7ea4102d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009900 tx=0x7f7e9402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.573+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1108079191 shutdown_connections 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.573+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1108079191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.573+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1108079191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea41082b0 0x7f7ea41086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.573+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1108079191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea4102980 0x7f7ea4102d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.573+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1108079191 >> 172.21.15.16:0/1108079191 conn(0x7f7ea40fe380 msgr2=0x7f7ea41007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:36.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.580+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1108079191 shutdown_connections 2024-09-21T02:18:36.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.581+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1108079191 wait complete. 2024-09-21T02:18:36.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.582+0000 7f7ea9fc3640 1 Processor -- start 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 -- start start 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4102980 0x7f7ea410c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 0x7f7ea410ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 0x7f7ea410ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:36.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 0x7f7ea410ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:46996/0 (socket says 172.21.15.16:46996) 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea2ffd640 1 -- 172.21.15.16:0/1239341904 learned_addr learned my addr 172.21.15.16:0/1239341904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4115a80 con 0x7f7ea4108c00 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4115900 con 0x7f7ea4102980 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4115c00 con 0x7f7ea41082b0 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.583+0000 7f7ea37fe640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4102980 0x7f7ea410c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:36.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.584+0000 7f7ea3fff640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.586+0000 7f7ea3fff640 1 -- 172.21.15.16:0/1239341904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 msgr2=0x7f7ea410ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.586+0000 7f7ea3fff640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 0x7f7ea410ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.586+0000 7f7ea3fff640 1 -- 172.21.15.16:0/1239341904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4102980 msgr2=0x7f7ea410c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.586+0000 7f7ea3fff640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4102980 0x7f7ea410c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.586+0000 7f7ea3fff640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea41a41a0 con 0x7f7ea4108c00 2024-09-21T02:18:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.587+0000 7f7ea3fff640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410d3c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009fd0 tx=0x7f7e9c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.590+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00ce20 con 0x7f7ea4108c00 2024-09-21T02:18:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.590+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e9c005020 con 0x7f7ea4108c00 2024-09-21T02:18:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.590+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c010690 con 0x7f7ea4108c00 2024-09-21T02:18:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.592+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea41a4490 con 0x7f7ea4108c00 2024-09-21T02:18:36.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.592+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea41a4960 con 0x7f7ea4108c00 2024-09-21T02:18:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.598+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea4103bf0 con 0x7f7ea4108c00 2024-09-21T02:18:36.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.599+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f7e9c004a40 con 0x7f7ea4108c00 2024-09-21T02:18:36.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.600+0000 7f7ea0ff9640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 0x7f7e8c0b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:36.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.600+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f7e9c0e7ec0 con 0x7f7ea4108c00 2024-09-21T02:18:36.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.603+0000 7f7ea37fe640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 0x7f7e8c0b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:36.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.604+0000 7f7ea37fe640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 0x7f7e8c0b8d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009230 tx=0x7f7e94005a20 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.612+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7e9c01d060 con 0x7f7ea4108c00 2024-09-21T02:18:36.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.992+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ea4111200 con 0x7f7ea4108c00 2024-09-21T02:18:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:36.998+0000 7f7ea0ff9640 1 -- 172.21.15.16:0/1239341904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+25379 (secure 0 0 0) 0x7f7e9c0adff0 con 0x7f7ea4108c00 2024-09-21T02:18:36.999 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:18:37.000 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:18:30.979507+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.001+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 msgr2=0x7f7e8c0b8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 0x7f7e8c0b8d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009230 tx=0x7f7e94005a20 comp rx=0 tx=0).stop 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 msgr2=0x7f7ea410d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410d3c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009fd0 tx=0x7f7e9c00ef10 comp rx=0 tx=0).stop 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 shutdown_connections 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f7e8c0b6850 0x7f7e8c0b8d60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea4108c00 0x7f7ea410d3c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ea41082b0 0x7f7ea410ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 --2- 172.21.15.16:0/1239341904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ea4102980 0x7f7ea410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 >> 172.21.15.16:0/1239341904 conn(0x7f7ea40fe380 msgr2=0x7f7ea41007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 shutdown_connections 2024-09-21T02:18:37.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.002+0000 7f7ea9fc3640 1 -- 172.21.15.16:0/1239341904 wait complete. 2024-09-21T02:18:37.023 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T01:59:52.388767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-21T02:04:29.848315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-21T02:04:32.294267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-21T02:04:38.676726+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-21T02:18:37.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.231+0000 7f684c38b640 1 -- 172.21.15.16:0/2838088885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6844108c10 msgr2=0x7f684406a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.231+0000 7f684c38b640 1 --2- 172.21.15.16:0/2838088885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6844108c10 0x7f684406a720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f683c009be0 tx=0x7f683c02a6f0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.232+0000 7f684c38b640 1 -- 172.21.15.16:0/2838088885 shutdown_connections 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.232+0000 7f684c38b640 1 --2- 172.21.15.16:0/2838088885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6844108c10 0x7f684406a720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.232+0000 7f684c38b640 1 --2- 172.21.15.16:0/2838088885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68441082c0 0x7f68441086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.232+0000 7f684c38b640 1 --2- 172.21.15.16:0/2838088885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 0x7f6844102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.232+0000 7f684c38b640 1 -- 172.21.15.16:0/2838088885 >> 172.21.15.16:0/2838088885 conn(0x7f68440fe3b0 msgr2=0x7f6844100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.237+0000 7f684c38b640 1 -- 172.21.15.16:0/2838088885 shutdown_connections 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.237+0000 7f684c38b640 1 -- 172.21.15.16:0/2838088885 wait complete. 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.238+0000 7f684c38b640 1 Processor -- start 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 -- start start 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 0x7f6844074fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 0x7f684406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 0x7f684406e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6844115be0 con 0x7f6844108c10 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6844115a60 con 0x7f6844102990 2024-09-21T02:18:37.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.239+0000 7f684c38b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6844115d60 con 0x7f68441082c0 2024-09-21T02:18:37.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.240+0000 7f6849926640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 0x7f684406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.240+0000 7f6849926640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 0x7f684406e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:47000/0 (socket says 172.21.15.16:47000) 2024-09-21T02:18:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.240+0000 7f6849926640 1 -- 172.21.15.16:0/1744861364 learned_addr learned my addr 172.21.15.16:0/1744861364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 0x7f684406e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a127640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 0x7f6844074fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 -- 172.21.15.16:0/1744861364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 msgr2=0x7f684406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 0x7f684406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 -- 172.21.15.16:0/1744861364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 msgr2=0x7f6844074fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 0x7f6844074fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.241+0000 7f684a928640 1 -- 172.21.15.16:0/1744861364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6844073c30 con 0x7f6844108c10 2024-09-21T02:18:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.246+0000 7f684a928640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 0x7f684406e990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f683c034040 tx=0x7f683c031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.252+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c041020 con 0x7f6844108c10 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.252+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f683c030cc0 con 0x7f6844108c10 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.252+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c036040 con 0x7f6844108c10 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.254+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6844073ec0 con 0x7f6844108c10 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.254+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6844074330 con 0x7f6844108c10 2024-09-21T02:18:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.255+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6844103c00 con 0x7f6844108c10 2024-09-21T02:18:37.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.260+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f683c044020 con 0x7f6844108c10 2024-09-21T02:18:37.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.261+0000 7f683b7fe640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 0x7f68300b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.262+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f683c108570 con 0x7f6844108c10 2024-09-21T02:18:37.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.266+0000 7f684a127640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 0x7f68300b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.267+0000 7f684a127640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 0x7f68300b8d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68400058c0 tx=0x7f684000a2d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:37 smithi123 ceph-mon[33453]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:37 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1239341904' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:18:37.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.271+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f683c04b070 con 0x7f6844108c10 2024-09-21T02:18:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:37 smithi016 ceph-mon[31245]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:37.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:37 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1239341904' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:18:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:37 smithi161 ceph-mon[34061]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:37.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:37 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1239341904' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:18:37.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.638+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6844074650 con 0x7f6844108c10 2024-09-21T02:18:37.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.639+0000 7f683b7fe640 1 -- 172.21.15.16:0/1744861364 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f683c0ce6a0 con 0x7f6844108c10 2024-09-21T02:18:37.640 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-09-21T02:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.643+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 msgr2=0x7f68300b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.643+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 0x7f68300b8d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68400058c0 tx=0x7f684000a2d0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.643+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 msgr2=0x7f684406e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.643+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 0x7f684406e990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f683c034040 tx=0x7f683c031670 comp rx=0 tx=0).stop 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 shutdown_connections 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f68300b6880 0x7f68300b8d90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6844108c10 0x7f684406e990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68441082c0 0x7f684406e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 --2- 172.21.15.16:0/1744861364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6844102990 0x7f6844074fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.644+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 >> 172.21.15.16:0/1744861364 conn(0x7f68440fe3b0 msgr2=0x7f684410f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.645+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 shutdown_connections 2024-09-21T02:18:37.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.645+0000 7f684c38b640 1 -- 172.21.15.16:0/1744861364 wait complete. 2024-09-21T02:18:37.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-21T02:18:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.842+0000 7fdb269eb640 1 -- 172.21.15.16:0/59697650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2010a0c0 msgr2=0x7fdb2010a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.842+0000 7fdb269eb640 1 --2- 172.21.15.16:0/59697650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2010a0c0 0x7fdb2010a4d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009ca0 tx=0x7fdb1802a8f0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.848+0000 7fdb269eb640 1 -- 172.21.15.16:0/59697650 shutdown_connections 2024-09-21T02:18:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.848+0000 7fdb269eb640 1 --2- 172.21.15.16:0/59697650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb20110340 0x7fdb20116b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.848+0000 7fdb269eb640 1 --2- 172.21.15.16:0/59697650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2010fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.848+0000 7fdb269eb640 1 --2- 172.21.15.16:0/59697650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb2010a0c0 0x7fdb2010a4d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.848+0000 7fdb269eb640 1 -- 172.21.15.16:0/59697650 >> 172.21.15.16:0/59697650 conn(0x7fdb200795d0 msgr2=0x7fdb201081e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.854+0000 7fdb269eb640 1 -- 172.21.15.16:0/59697650 shutdown_connections 2024-09-21T02:18:37.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.855+0000 7fdb269eb640 1 -- 172.21.15.16:0/59697650 wait complete. 2024-09-21T02:18:37.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 Processor -- start 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 -- start start 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 0x7fdb2007d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb20119ef0 con 0x7fdb20110340 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb20119d70 con 0x7fdb2010a0c0 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.856+0000 7fdb269eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2011a070 con 0x7fdb2010f9f0 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb1f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb1ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb1ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:57100/0 (socket says 172.21.15.16:57100) 2024-09-21T02:18:37.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb1ffff640 1 -- 172.21.15.16:0/1039514845 learned_addr learned my addr 172.21.15.16:0/1039514845 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb24f88640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 0x7fdb2007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.858+0000 7fdb1ffff640 1 -- 172.21.15.16:0/1039514845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 msgr2=0x7fdb2007d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.857+0000 7fdb1f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.16:48042/0 (socket says 172.21.15.16:48042) 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb1ffff640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb1ffff640 1 -- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 msgr2=0x7fdb2007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb1ffff640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 0x7fdb2007d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb1ffff640 1 -- 172.21.15.16:0/1039514845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb200829b0 con 0x7fdb2010a0c0 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb24f88640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 0x7fdb2007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.859+0000 7fdb1f7fe640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T02:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.860+0000 7fdb1ffff640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb180316b0 tx=0x7fdb180316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:37.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.860+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18031d10 con 0x7fdb2010a0c0 2024-09-21T02:18:37.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.860+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb1802d070 con 0x7fdb2010a0c0 2024-09-21T02:18:37.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.860+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1803b660 con 0x7fdb2010a0c0 2024-09-21T02:18:37.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.862+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb20082c40 con 0x7fdb2010a0c0 2024-09-21T02:18:37.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.862+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb20083140 con 0x7fdb2010a0c0 2024-09-21T02:18:37.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.864+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2010b330 con 0x7fdb2010a0c0 2024-09-21T02:18:37.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.869+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fdb18004b40 con 0x7fdb2010a0c0 2024-09-21T02:18:37.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.870+0000 7fdb1d7fa640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 0x7fdb080b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:37.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.870+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7fdb18108f80 con 0x7fdb2010a0c0 2024-09-21T02:18:37.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.871+0000 7fdb1f7fe640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 0x7fdb080b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:37.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.872+0000 7fdb1f7fe640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 0x7fdb080b8d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdb2010c0c0 tx=0x7fdb14005f90 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:37.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:37.876+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdb180cf130 con 0x7fdb2010a0c0 2024-09-21T02:18:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.106+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdb201120c0 con 0x7fdb2010a0c0 2024-09-21T02:18:38.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.106+0000 7fdb1d7fa640 1 -- 172.21.15.16:0/1039514845 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7fdb180d3510 con 0x7fdb2010a0c0 2024-09-21T02:18:38.107 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-09-21T02:18:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 msgr2=0x7fdb080b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 0x7fdb080b8d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdb2010c0c0 tx=0x7fdb14005f90 comp rx=0 tx=0).stop 2024-09-21T02:18:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 msgr2=0x7fdb20083d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb180316b0 tx=0x7fdb180316e0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 shutdown_connections 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fdb080b6880 0x7fdb080b8d90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20110340 0x7fdb2007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb2010f9f0 0x7fdb2007d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 --2- 172.21.15.16:0/1039514845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb2010a0c0 0x7fdb20083d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 >> 172.21.15.16:0/1039514845 conn(0x7fdb200795d0 msgr2=0x7fdb20079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 shutdown_connections 2024-09-21T02:18:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.109+0000 7fdb269eb640 1 -- 172.21.15.16:0/1039514845 wait complete. 2024-09-21T02:18:38.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/4117965870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e881061c0 msgr2=0x7f4e881065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/4117965870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e881061c0 0x7f4e881065d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009900 tx=0x7f4e7c02a410 comp rx=0 tx=0).stop 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/4117965870 shutdown_connections 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/4117965870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 0x7f4e881105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/4117965870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e881061c0 0x7f4e881065d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/4117965870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e88101fe0 0x7f4e881023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/4117965870 >> 172.21.15.16:0/4117965870 conn(0x7f4e88079e10 msgr2=0x7f4e8807a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/4117965870 shutdown_connections 2024-09-21T02:18:38.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.238+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/4117965870 wait complete. 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 Processor -- start 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 -- start start 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 0x7f4e8819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 0x7f4e8819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113890 con 0x7f4e88101fe0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113710 con 0x7f4e881061c0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8ca4d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88113a10 con 0x7f4e88106b10 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60912/0 (socket says 172.21.15.16:60912) 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e8659c640 1 -- 172.21.15.16:0/3603137773 learned_addr learned my addr 172.21.15.16:0/3603137773 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.239+0000 7f4e85d9b640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 0x7f4e8819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e86d9d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 0x7f4e8819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 -- 172.21.15.16:0/3603137773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 msgr2=0x7f4e8819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 0x7f4e8819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 -- 172.21.15.16:0/3603137773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 msgr2=0x7f4e8819a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 0x7f4e8819a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 -- 172.21.15.16:0/3603137773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e881a0280 con 0x7f4e88101fe0 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e85d9b640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 0x7f4e8819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:18:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8659c640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800e9c0 tx=0x7f4e7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7800cd80 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e780044b0 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78010430 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e881a0570 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.240+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e881a6d00 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.241+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e881023f0 con 0x7f4e88101fe0 2024-09-21T02:18:38.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.243+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f4e78003f00 con 0x7f4e88101fe0 2024-09-21T02:18:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.243+0000 7f4e777fe640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 0x7f4e700b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.244+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f4e780e8160 con 0x7f4e88101fe0 2024-09-21T02:18:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.244+0000 7f4e85d9b640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 0x7f4e700b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.244+0000 7f4e85d9b640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 0x7f4e700b8b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0057c0 tx=0x7f4e7c009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.247+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4e780ae320 con 0x7f4e88101fe0 2024-09-21T02:18:38.316 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:38 smithi161 ceph-mon[34061]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:38.316 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:38 smithi161 ceph-mon[34061]: pgmap v708: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 17 MiB/s wr, 445 op/s 2024-09-21T02:18:38.316 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:38 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1744861364' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:18:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:38 smithi016 ceph-mon[31245]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:38 smithi016 ceph-mon[31245]: pgmap v708: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 17 MiB/s wr, 445 op/s 2024-09-21T02:18:38.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:38 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1744861364' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:18:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.437+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4e8802e640 con 0x7f4e88101fe0 2024-09-21T02:18:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.437+0000 7f4e777fe640 1 -- 172.21.15.16:0/3603137773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f4e780b2700 con 0x7f4e88101fe0 2024-09-21T02:18:38.438 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-09-21T02:18:38.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 msgr2=0x7f4e700b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 0x7f4e700b8b90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0057c0 tx=0x7f4e7c009680 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 msgr2=0x7f4e8819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800e9c0 tx=0x7f4e7800ee80 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 shutdown_connections 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f4e700b6680 0x7f4e700b8b90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e88106b10 0x7f4e8819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e881061c0 0x7f4e8819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 --2- 172.21.15.16:0/3603137773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e88101fe0 0x7f4e8819a350 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.439+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 >> 172.21.15.16:0/3603137773 conn(0x7f4e88079e10 msgr2=0x7f4e881095c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.440+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 shutdown_connections 2024-09-21T02:18:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.440+0000 7f4e8ca4d640 1 -- 172.21.15.16:0/3603137773 wait complete. 2024-09-21T02:18:38.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-21T02:18:38.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:38 smithi123 ceph-mon[33453]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:38.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:38 smithi123 ceph-mon[33453]: pgmap v708: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 17 MiB/s wr, 445 op/s 2024-09-21T02:18:38.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:38 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1744861364' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 -- 172.21.15.16:0/1940135949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 msgr2=0x7fc81010d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 --2- 172.21.15.16:0/1940135949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81010d880 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc804009900 tx=0x7fc80402a410 comp rx=0 tx=0).stop 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 -- 172.21.15.16:0/1940135949 shutdown_connections 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 --2- 172.21.15.16:0/1940135949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 0x7fc810110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 --2- 172.21.15.16:0/1940135949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81010d880 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 --2- 172.21.15.16:0/1940135949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 0x7fc810104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 -- 172.21.15.16:0/1940135949 >> 172.21.15.16:0/1940135949 conn(0x7fc810079630 msgr2=0x7fc810079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 -- 172.21.15.16:0/1940135949 shutdown_connections 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.567+0000 7fc81548b640 1 -- 172.21.15.16:0/1940135949 wait complete. 2024-09-21T02:18:38.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.568+0000 7fc81548b640 1 Processor -- start 2024-09-21T02:18:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.568+0000 7fc81548b640 1 -- start start 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.568+0000 7fc81548b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 0x7fc81019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.568+0000 7fc81548b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.568+0000 7fc81548b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 0x7fc81019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc81548b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc810113840 con 0x7fc810105240 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc81548b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc8101136c0 con 0x7fc81010ddc0 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc81548b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8101139c0 con 0x7fc810104810 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60934/0 (socket says 172.21.15.16:60934) 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 -- 172.21.15.16:0/3247492893 learned_addr learned my addr 172.21.15.16:0/3247492893 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80effd640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 0x7fc81019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80f7fe640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 0x7fc81019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 -- 172.21.15.16:0/3247492893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 msgr2=0x7fc81019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 0x7fc81019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 -- 172.21.15.16:0/3247492893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 msgr2=0x7fc81019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 0x7fc81019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 -- 172.21.15.16:0/3247492893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8101a0390 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc80e7fc640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc804005940 tx=0x7fc8040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80403f030 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8101a0620 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc804003c90 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804035070 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.569+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8101a6e10 con 0x7fc810105240 2024-09-21T02:18:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.570+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc810104c20 con 0x7fc810105240 2024-09-21T02:18:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.572+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fc804042030 con 0x7fc810105240 2024-09-21T02:18:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.573+0000 7fc7f7fff640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 0x7fc7f80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.573+0000 7fc80effd640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 0x7fc7f80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.573+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7fc8041083b0 con 0x7fc810105240 2024-09-21T02:18:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.573+0000 7fc80effd640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 0x7fc7f80b8b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc800005fd0 tx=0x7fc800009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.576+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc8040d28c0 con 0x7fc810105240 2024-09-21T02:18:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.768+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc810108ef0 con 0x7fc810105240 2024-09-21T02:18:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.768+0000 7fc7f7fff640 1 -- 172.21.15.16:0/3247492893 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v98) ==== 103+0+11 (secure 0 0 0) 0x7fc8040d7800 con 0x7fc810105240 2024-09-21T02:18:38.769 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 msgr2=0x7fc7f80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 0x7fc7f80b8b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc800005fd0 tx=0x7fc800009680 comp rx=0 tx=0).stop 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 msgr2=0x7fc81019aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc804005940 tx=0x7fc8040042f0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 shutdown_connections 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fc7f80b6680 0x7fc7f80b8b90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc81010ddc0 0x7fc81019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc810105240 0x7fc81019aa00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 --2- 172.21.15.16:0/3247492893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc810104810 0x7fc81019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 >> 172.21.15.16:0/3247492893 conn(0x7fc810079630 msgr2=0x7fc810100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 shutdown_connections 2024-09-21T02:18:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.771+0000 7fc81548b640 1 -- 172.21.15.16:0/3247492893 wait complete. 2024-09-21T02:18:38.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-21T02:18:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 -- 172.21.15.16:0/3360742789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c104900 msgr2=0x7f358c104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 --2- 172.21.15.16:0/3360742789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c104900 0x7f358c104d10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f357c009900 tx=0x7f357c02a410 comp rx=0 tx=0).stop 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 -- 172.21.15.16:0/3360742789 shutdown_connections 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 --2- 172.21.15.16:0/3360742789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 0x7f358c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 --2- 172.21.15.16:0/3360742789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c105250 0x7f358c10d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 --2- 172.21.15.16:0/3360742789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c104900 0x7f358c104d10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 -- 172.21.15.16:0/3360742789 >> 172.21.15.16:0/3360742789 conn(0x7f358c0fe3b0 msgr2=0x7f358c100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.896+0000 7f359104d640 1 -- 172.21.15.16:0/3360742789 shutdown_connections 2024-09-21T02:18:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.897+0000 7f359104d640 1 -- 172.21.15.16:0/3360742789 wait complete. 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.897+0000 7f359104d640 1 Processor -- start 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 -- start start 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c104900 0x7f358c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 0x7f358c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f358c113870 con 0x7f358c105250 2024-09-21T02:18:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f358c1136f0 con 0x7f358c10dd60 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f359104d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f358c1139f0 con 0x7f358c104900 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60944/0 (socket says 172.21.15.16:60944) 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 -- 172.21.15.16:0/2831468068 learned_addr learned my addr 172.21.15.16:0/2831468068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358ad9d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c104900 0x7f358c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358b59e640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 0x7f358c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 -- 172.21.15.16:0/2831468068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c104900 msgr2=0x7f358c19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c104900 0x7f358c19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 -- 172.21.15.16:0/2831468068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 msgr2=0x7f358c19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 0x7f358c19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 -- 172.21.15.16:0/2831468068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358c1a0300 con 0x7f358c105250 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.898+0000 7f358a59c640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f358000d950 tx=0x7f358000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.899+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580017070 con 0x7f358c105250 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.899+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f358c1a05f0 con 0x7f358c105250 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.899+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3580009c30 con 0x7f358c105250 2024-09-21T02:18:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.899+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35800047a0 con 0x7f358c105250 2024-09-21T02:18:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.899+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f358c1a6d80 con 0x7f358c105250 2024-09-21T02:18:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.900+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f358c104d10 con 0x7f358c105250 2024-09-21T02:18:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.902+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f3580004020 con 0x7f358c105250 2024-09-21T02:18:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.903+0000 7f3573fff640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 0x7f35740b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:18:38.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.903+0000 7f358ad9d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 0x7f35740b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:18:38.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.903+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9482+0+0 (secure 0 0 0) 0x7f35800e7d60 con 0x7f358c105250 2024-09-21T02:18:38.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.903+0000 7f358ad9d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 0x7f35740b8b90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f357c000c00 tx=0x7f357c009680 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:18:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:38.906+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3580020020 con 0x7f358c105250 2024-09-21T02:18:39.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.146+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f358c1088c0 con 0x7f358c105250 2024-09-21T02:18:39.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.155+0000 7f3573fff640 1 -- 172.21.15.16:0/2831468068 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v24) ==== 81+0+834054 (secure 0 0 0) 0x7f35800ade90 con 0x7f358c105250 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 24, 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 14223, 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-09-21T02:18:39.161 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3647438626 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3647438626 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.162 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/3647438626", 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-09-21T01:57:51.923134+0000", 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540719139924082687, 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24187, 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540719139924082687, 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.163 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.165 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-21T02:18:39.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-21T02:18:39.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:39.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T02:18:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-21T02:18:39.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.188 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-21T02:18:39.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-21T02:18:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-21T02:18:39.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-21T02:18:39.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-21T02:18:39.210 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-21T02:18:39.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-21T02:18:39.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T02:18:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.223 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-21T02:18:39.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T02:18:39.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba_metrics": { 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba_metrics", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T02:18:39.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-21T02:18:39.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.242 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.246 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T02:18:39.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.251 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T02:18:39.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.258 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-21T02:18:39.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_count_max": { 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_interval": { 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_interval", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T02:18:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.273 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-21T02:18:39.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.278 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.279 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T02:18:39.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T02:18:39.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T02:18:39.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T02:18:39.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T02:18:39.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T02:18:39.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.315 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T02:18:39.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.326 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.327 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:39.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-21T02:18:39.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "sso_oauth2": { 2024-09-21T02:18:39.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sso_oauth2", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.341 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.343 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.350 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-21T02:18:39.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-21T02:18:39.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:39.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.363 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-21T02:18:39.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-21T02:18:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-21T02:18:39.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.371 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.372 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.374 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:39.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.381 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.383 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.386 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.391 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.392 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.401 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.404 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.406 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.408 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T02:18:39.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.414 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-21T02:18:39.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.417 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.429 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.432 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.436 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-21T02:18:39.441 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.443 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.444 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.445 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.446 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.448 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.449 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.452 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.463 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.469 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-21T02:18:39.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.476 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.479 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-21T02:18:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-21T02:18:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-21T02:18:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-21T02:18:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-21T02:18:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-21T02:18:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-21T02:18:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-21T02:18:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-21T02:18:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-21T02:18:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T02:18:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24200, 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540719139924082687, 2024-09-21T02:18:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-21T02:18:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-21T02:18:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-21T02:18:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-21T02:18:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-21T02:18:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-21T02:18:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-21T02:18:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-21T02:18:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-21T02:18:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-21T02:18:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-21T02:18:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-21T02:18:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T02:18:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-21T02:18:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-21T02:18:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T02:18:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-21T02:18:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-21T02:18:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba_metrics": { 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba_metrics", 2024-09-21T02:18:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-21T02:18:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-21T02:18:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-21T02:18:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T02:18:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-21T02:18:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_count_max": { 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_interval": { 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_interval", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T02:18:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T02:18:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T02:18:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T02:18:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T02:18:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T02:18:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T02:18:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-09-21T02:18:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-09-21T02:18:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.781 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1039514845' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.781 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3603137773' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.781 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:39 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3247492893' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1039514845' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3603137773' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:39 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3247492893' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1039514845' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3603137773' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.782 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:39 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3247492893' entity='client.admin' cmd={"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} : dispatch 2024-09-21T02:18:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T02:18:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T02:18:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T02:18:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T02:18:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T02:18:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-21T02:18:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "sso_oauth2": { 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sso_oauth2", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-21T02:18:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-21T02:18:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-21T02:18:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T02:18:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-21T02:18:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-21T02:18:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-21T02:18:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-21T02:18:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-21T02:18:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T02:18:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T02:18:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-21T02:18:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-09-21T02:18:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-21T02:18:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-21T02:18:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-21T02:18:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-21T02:18:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-21T02:18:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "restful", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_schedule" 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-09-21T02:18:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-21T02:18:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-09-21T02:18:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-21T02:18:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-09-21T02:18:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-21T02:18:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-21T02:18:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-09-21T02:18:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:10.4.8", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-09-21T02:18:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nginx": { 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nginx", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nginx container image", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_oauth2_proxy": { 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_oauth2_proxy", 2024-09-21T02:18:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "oauth2-proxy container image", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba_metrics": { 2024-09-21T02:18:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba_metrics", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-09-21T02:18:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-09-21T02:18:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-09-21T02:18:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-09-21T02:18:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-21T02:18:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-21T02:18:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_count_max": { 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_count_max", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_keepalive_interval": { 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_keepalive_interval", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-09-21T02:18:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-09-21T02:18:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-21T02:18:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-21T02:18:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-21T02:18:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-21T02:18:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-21T02:18:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-09-21T02:18:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-09-21T02:18:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-09-21T02:18:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "sso_oauth2": { 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sso_oauth2", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-09-21T02:18:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-09-21T02:18:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-09-21T02:18:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-09-21T02:18:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-09-21T02:18:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-09-21T02:18:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-21T02:18:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-09-21T02:18:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-09-21T02:18:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-09-21T02:18:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-09-21T02:18:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-09-21T02:18:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max_requests": { 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_requests", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "secondary_zone_period_retry_limit": { 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-09-21T02:18:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-09-21T02:18:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-09-21T02:18:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-09-21T02:18:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smb", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "internal_store_backend": { 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "internal_store_backend", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "set internal store backend. for develoment and testing only", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "update_orchestration": { 2024-09-21T02:18:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "update_orchestration", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-09-21T02:18:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-09-21T02:18:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-21T02:18:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "sqlite3_killpoint": { 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sqlite3_killpoint", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "dev", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-09-21T02:18:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-21T02:18:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-21T02:18:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-09-21T02:18:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 5, 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3621112339 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2305818243 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4078492053 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3340959932 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2990957817 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1532412515 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-21T02:18:40.580 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 msgr2=0x7f35740b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 0x7f35740b8b90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f357c000c00 tx=0x7f357c009680 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 msgr2=0x7f358c19a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f358000d950 tx=0x7f358000de10 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 shutdown_connections 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f35740b6680 0x7f35740b8b90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f358c10dd60 0x7f358c19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f358c105250 0x7f358c19a970 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 --2- 172.21.15.16:0/2831468068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f358c104900 0x7f358c19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 >> 172.21.15.16:0/2831468068 conn(0x7f358c0fe3b0 msgr2=0x7f358c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 shutdown_connections 2024-09-21T02:18:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:18:39.161+0000 7f359104d640 1 -- 172.21.15.16:0/2831468068 wait complete. 2024-09-21T02:18:40.623 DEBUG:tasks.check_counter:Getting stats from a 2024-09-21T02:18:40.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.a perf dump 2024-09-21T02:18:40.623 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:40 smithi016 ceph-mon[31245]: pgmap v709: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 11 MiB/s wr, 407 op/s 2024-09-21T02:18:40.623 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:40 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2831468068' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T02:18:40.624 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:40 smithi123 ceph-mon[33453]: pgmap v709: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 11 MiB/s wr, 407 op/s 2024-09-21T02:18:40.624 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:40 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2831468068' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T02:18:40.624 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:40 smithi161 ceph-mon[34061]: pgmap v709: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 11 MiB/s wr, 407 op/s 2024-09-21T02:18:40.624 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:40 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2831468068' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json-pretty"} : dispatch 2024-09-21T02:18:40.897 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:41.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:41 smithi016 ceph-mon[31245]: pgmap v710: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 326 op/s 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1604, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1606, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 481317, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 510073, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.184496784, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.077213265, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.042231054, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.051634506, 2024-09-21T02:18:41.413 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1604, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.111927541, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000069780 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.414 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 8, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 180, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 4494, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 385790, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.020476735, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.014563795, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.004009227, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000065835, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 179, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.017272415, 2024-09-21T02:18:41.415 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000096493 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 384768 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 962, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 956, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1049806, 2024-09-21T02:18:41.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 369784, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.128669347, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.052384919, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.050530520, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.028084207, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 955, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.053130786, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000055634 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 875616, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 144640 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3817, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.120439221, 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000031553 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-09-21T02:18:41.420 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 1, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-09-21T02:18:41.421 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:41.422 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-09-21T02:18:41.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-09-21T02:18:41.424 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_write_failed": 0 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 6, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-09-21T02:18:41.425 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 2, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 2, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 4196308, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 4195917, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 6 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 3, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 3, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 1, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 1, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-09-21T02:18:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 116416, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 316420096 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.428 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.429 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-09-21T02:18:41.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.432 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:41 smithi161 ceph-mon[34061]: pgmap v710: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 326 op/s 2024-09-21T02:18:41.432 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-09-21T02:18:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-09-21T02:18:41.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-09-21T02:18:41.439 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 3, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 1, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 1813, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:41.440 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:41.441 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:41.442 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 197463, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 54, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 195, 2024-09-21T02:18:41.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 8672, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 13, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 2315, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-09-21T02:18:41.444 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 2315, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2315, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2.044966262, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000883354 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2315, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2316 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 2315, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 2315, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-09-21T02:18:41.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 1506, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 807, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-09-21T02:18:41.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-09-21T02:18:41.447 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 98, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 14, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 6, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 6, 2024-09-21T02:18:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:41.449 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2570, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1288360, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2570, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2570, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1288360, 2024-09-21T02:18:41.450 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 2315, 2024-09-21T02:18:41.451 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 2009, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2315, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 2009, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:41.452 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 2315, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 2315, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2315, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 2315, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:41.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x5592682536c0": { 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:41.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:41.456 INFO:teuthology.orchestra.run.smithi016.stdout: "trackedop": { 2024-09-21T02:18:41.456 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-09-21T02:18:41.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:41.456 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:41.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:41 smithi123 ceph-mon[33453]: pgmap v710: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 326 op/s 2024-09-21T02:18:43.531 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:18:43.532 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:18:43.532 DEBUG:tasks.check_counter:Getting stats from d 2024-09-21T02:18:43.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.d perf dump 2024-09-21T02:18:43.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:43 smithi123 ceph-mon[33453]: pgmap v711: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-09-21T02:18:43.799 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:43.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:43 smithi016 ceph-mon[31245]: pgmap v711: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-09-21T02:18:43.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:43 smithi161 ceph-mon[34061]: pgmap v711: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-09-21T02:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 240, 2024-09-21T02:18:45.333 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 235, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 842912, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 145184, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.047067422, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.021687420, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.033220003, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 234, 2024-09-21T02:18:45.334 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.015420537, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000065899 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 842912, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 145184 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 5, 2024-09-21T02:18:45.335 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 182, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3104, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 112320, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.042587313, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.036808900, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.003648934, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 181, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.013337680, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000073688 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.336 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 112320 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-09-21T02:18:45.337 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.000404577, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000282103, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:45.338 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:45.339 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:45.340 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-21T02:18:45.341 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148967, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-21T02:18:45.342 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 968, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-21T02:18:45.343 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 241, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 808977, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 241, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 241, 2024-09-21T02:18:45.344 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 808977, 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:45.345 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:45.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:45 smithi016 ceph-mon[31245]: pgmap v712: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 413 KiB/s wr, 230 op/s 2024-09-21T02:18:45.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:45 smithi123 ceph-mon[33453]: pgmap v712: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 413 KiB/s wr, 230 op/s 2024-09-21T02:18:45.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:45 smithi161 ceph-mon[34061]: pgmap v712: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 413 KiB/s wr, 230 op/s 2024-09-21T02:18:47.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:47 smithi016 ceph-mon[31245]: pgmap v713: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 168 op/s 2024-09-21T02:18:47.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:47 smithi123 ceph-mon[33453]: pgmap v713: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 168 op/s 2024-09-21T02:18:47.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:47 smithi161 ceph-mon[34061]: pgmap v713: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 168 op/s 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-21T02:18:48.597 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-21T02:18:48.597 DEBUG:tasks.check_counter:Getting stats from g 2024-09-21T02:18:48.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.g perf dump 2024-09-21T02:18:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:49.415 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 806, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 808, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 270761, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 256309, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.119035443, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.053523426, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.025857309, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.030988207, 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:49.416 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 806, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.073374441, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000091035 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:49.417 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1525, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1701, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 390012, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 871009, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.157152186, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.068433941, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.032127439, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.047068789, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1699, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.115442315, 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000067947 2024-09-21T02:18:49.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 386320 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 264, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 256, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 880724, 2024-09-21T02:18:49.419 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 147973, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.046417630, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.021802393, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.031545011, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000061799, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 256, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.021444676, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000083768 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 879376, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 147008 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-09-21T02:18:49.421 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3773, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.123417895, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000032710 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:49.423 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 1, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-09-21T02:18:49.424 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:49.425 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-09-21T02:18:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-09-21T02:18:49.428 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_started": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "uninline_write_failed": 0 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-09-21T02:18:49.429 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 6, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 2, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 2, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 4196308, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 4195917, 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-09-21T02:18:49.430 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 6 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 3, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 3, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 0, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 1, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 1, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 0, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 0, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 0, 2024-09-21T02:18:49.431 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 117328, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 316424192 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-09-21T02:18:49.432 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-09-21T02:18:49.433 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.434 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.436 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.437 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.439 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.441 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.442 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 3, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 1, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 1819, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:49.444 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:49.445 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:49.446 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 197497, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 56, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 195, 2024-09-21T02:18:49.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 8672, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 13, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 2333, 2024-09-21T02:18:49.448 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 2333, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2333, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.361928029, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001869664 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2333, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2334 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 2333, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 2333, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-09-21T02:18:49.449 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 1518, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 813, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:49.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-09-21T02:18:49.451 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 98, 2024-09-21T02:18:49.452 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 14, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 6, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 6, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-09-21T02:18:49.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:49.454 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 2591, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1292194, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 2591, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2591, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1292194, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 2333, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 2009, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2333, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 2009, 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:49.456 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 2333, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 2333, 2024-09-21T02:18:49.457 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 2333, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 2333, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:49.458 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x5647384bf2b0": { 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:49.459 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "trackedop": { 2024-09-21T02:18:49.460 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-09-21T02:18:49.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:49.461 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:49.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:49 smithi016 ceph-mon[31245]: pgmap v714: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 154 op/s 2024-09-21T02:18:49.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:49 smithi123 ceph-mon[33453]: pgmap v714: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 154 op/s 2024-09-21T02:18:49.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:49 smithi161 ceph-mon[34061]: pgmap v714: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.7 KiB/s wr, 154 op/s 2024-09-21T02:18:50.265 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:18:50.266 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:18:50.266 DEBUG:tasks.check_counter:Getting stats from j 2024-09-21T02:18:50.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.j perf dump 2024-09-21T02:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:18:51.056 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001317169, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000315460, 2024-09-21T02:18:51.057 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 5, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 184, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3104, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 112672, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.020111099, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.014268485, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.003988722, 2024-09-21T02:18:51.059 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 183, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.016868229, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000092176 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 112672 2024-09-21T02:18:51.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 244, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 239, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 844064, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 147264, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.050422023, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.020465365, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.038157433, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:51.061 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 238, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.017307661, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000072721 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 844064, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 147264 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-09-21T02:18:51.062 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:51.063 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:51.064 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:51.065 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148967, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-09-21T02:18:51.066 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 968, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:51.067 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 245, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 809561, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 245, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 245, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 809561, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-09-21T02:18:51.068 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-09-21T02:18:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:51.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-21T02:18:51.069 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-21T02:18:51.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:51 smithi016 ceph-mon[31245]: pgmap v715: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 94 op/s 2024-09-21T02:18:51.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:51 smithi123 ceph-mon[33453]: pgmap v715: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 94 op/s 2024-09-21T02:18:51.779 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-21T02:18:51.779 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-21T02:18:51.779 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-09-21T02:18:51.779 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-09-21T02:18:51.780 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-21T02:18:51.780 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-21T02:18:51.780 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-21T02:18:51.780 DEBUG:tasks.check_counter:Getting stats from b 2024-09-21T02:18:51.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.b perf dump 2024-09-21T02:18:51.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:51 smithi161 ceph-mon[34061]: pgmap v715: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 94 op/s 2024-09-21T02:18:52.044 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 3975, 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 3983, 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 1260206, 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 6048389, 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 60, 2024-09-21T02:18:52.617 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 8, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.431762789, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.152523942, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.181268714, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.101369054, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3978, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.947245628, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000238121 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 4969, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001053633, 2024-09-21T02:18:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000212 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 59168, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 32320 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 2415, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 2576, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 1613961, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 1059201, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 11, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.263422091, 2024-09-21T02:18:52.619 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.122550968, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.120544904, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.034017805, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2578, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.992328761, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000384921 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1917, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000428981, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000223 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.620 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 914720, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 155488 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 3556, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 3461, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 1375148, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 1639886, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 16, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 7, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.399533992, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.165428243, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.188138016, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.066333752, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3460, 2024-09-21T02:18:52.621 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.694875877, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000200831 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 4828, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001037539, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000214 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 349376, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 509648 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-09-21T02:18:52.622 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 659, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.278595386, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000422754 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2524, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.098522205, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000039034 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.623 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 503, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 503, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 503, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 3.100408237, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.006163833 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 0, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 21, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 1, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 160, 2024-09-21T02:18:52.624 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 59, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 101, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 75, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 3, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 3, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 2, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 855, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 837, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-09-21T02:18:52.625 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 138, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 4, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 234, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 6, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:52.626 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 329, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 479, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 498, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 223, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 2, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 83 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-09-21T02:18:52.627 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 28, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 56, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 4, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 22, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 18, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-09-21T02:18:52.628 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 11, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 342, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 1, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_started": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_write_failed": 0 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 530, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-09-21T02:18:52.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 530, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 2, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 2, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 2, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 8107125, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 0, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 294, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 1.003062671, 2024-09-21T02:18:52.630 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.003411777 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 0 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 135, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 153, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 18, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 29, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 30, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 1, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 160, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 178, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 18, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 3, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 37, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 34, 2024-09-21T02:18:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 125624, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 340541440 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 648, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 1, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 503, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 29, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 2, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 22, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.022316101, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001014368 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 167, 2024-09-21T02:18:52.632 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.454225756, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002719914 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 229, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 2.370132293, 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.010349922 2024-09-21T02:18:52.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 11, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001663646, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000151240 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 10, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.016874192, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001687419 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.636 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 11, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.081158778, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.007378070 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000472406, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000472406 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 11, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.013931913, 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001266537 2024-09-21T02:18:52.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.002307510, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001153755 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.638 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 4, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.049497983, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.012374495 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 5, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.032357205, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.006471441 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.639 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001470023, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000735011 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:52.640 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 10, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.024500764, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002450076 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.641 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 18, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.029499667, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.001638870 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 4, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 2, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 117, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 58, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 677, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:52.643 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:52.644 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-09-21T02:18:52.645 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 350999, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 122, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:52.646 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 195, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 521786, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 3418, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 3508, 2024-09-21T02:18:52.647 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 4048801, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 37, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 3508, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3508, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 88.847890496, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.025327220 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3508, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 3566 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 3508, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 1, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 3507, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-09-21T02:18:52.648 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 11, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 1, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 295, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 48, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 1, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 3108, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 0, 2024-09-21T02:18:52.649 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 22, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-09-21T02:18:52.650 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 98, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 22, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 12, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 12, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-09-21T02:18:52.651 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 67, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 13 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 1, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 522, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-09-21T02:18:52.652 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 5, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 2596, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 18, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 9942, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 3292869, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 9942, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 9942, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 3292869, 2024-09-21T02:18:52.653 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 3508, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 4051627, 2024-09-21T02:18:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 3508, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 4051627, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:52.655 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 3508, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 3508, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 3508, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 3508, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 18, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 2826, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:52.656 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 18, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 2826, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x560423e79520": { 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:52.657 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 530, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 3912821, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 530, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 294, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 3912821, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:52.658 INFO:teuthology.orchestra.run.smithi123.stdout: "trackedop": { 2024-09-21T02:18:52.659 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 319 2024-09-21T02:18:52.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:52.659 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-21T02:18:53.418 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-09-21T02:18:53.418 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-09-21T02:18:53.419 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=28 2024-09-21T02:18:53.419 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-09-21T02:18:53.419 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-09-21T02:18:53.419 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-09-21T02:18:53.419 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-09-21T02:18:53.419 DEBUG:tasks.check_counter:Getting stats from e 2024-09-21T02:18:53.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.e perf dump 2024-09-21T02:18:53.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:53 smithi123 ceph-mon[33453]: pgmap v716: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:18:53.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:53 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:18:53.684 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:18:53.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:53 smithi016 ceph-mon[31245]: pgmap v716: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:18:53.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:53 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:18:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:53 smithi161 ceph-mon[34061]: pgmap v716: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:18:53.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:53 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T02:18:54.191 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-21T02:18:54.191 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1148, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 974, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 368389, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 424610, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 10, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 5, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.121642763, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.060704806, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.063451974, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.004573232, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 972, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.059208718, 2024-09-21T02:18:54.192 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000060914 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1999, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000493601, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000246 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 84, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 280, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 28542, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 448608, 2024-09-21T02:18:54.193 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 13, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 6, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.024802146, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.013895479, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.009263127, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000324747, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 278, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.026833439, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000096523 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 146, 2024-09-21T02:18:54.194 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000033612, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000230 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 391040 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 374, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 379, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 922193, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 241925, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 12, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 8, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.058539907, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.025614040, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.047249923, 2024-09-21T02:18:54.195 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000135557, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 379, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.043586367, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000115003 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 137, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000038505, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000281 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 897520, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 160880 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-09-21T02:18:54.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 49, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001680297, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000034291 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000040703, 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000020351 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 1, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 1, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 11, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 0, 2024-09-21T02:18:54.198 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 16, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 16, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 13, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 8, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 3, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 36, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 29, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 1, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 6, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:54.199 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 4, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 234, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:54.200 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 0 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-09-21T02:18:54.201 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 20, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 13, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 14, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:54.202 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 11, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 105, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_started": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_write_failed": 0 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-09-21T02:18:54.203 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 6, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 6, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 2, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 2, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 2, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 4196308, 2024-09-21T02:18:54.204 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 0, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 5, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.010002086, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002000417 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 0 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 20, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 24, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 4, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 19, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 33, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 14, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 16, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 24, 2024-09-21T02:18:54.205 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 8, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 119304, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 325857280 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 1, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 1, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 23, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.206 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-09-21T02:18:54.207 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.208 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.210 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.212 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.214 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-09-21T02:18:54.215 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:54.216 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 3, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 1, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 0, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 0, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 677, 2024-09-21T02:18:54.217 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:54.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:54.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 271607, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 99, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:54.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 195, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 82521, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 359, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-09-21T02:18:54.221 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 41, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 12306, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 33, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 41, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 41, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.232994335, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.005682788 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 41, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 63 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.222 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 41, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 41, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 10, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 5, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 7, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 0, 2024-09-21T02:18:54.223 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 6, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 20, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:54.224 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-09-21T02:18:54.225 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 98, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 14, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 9, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 9, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 14, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 1 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 1, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-09-21T02:18:54.226 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 1602, 2024-09-21T02:18:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 1156509, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 1602, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1602, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1156509, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:54.228 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 41, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 12306, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 41, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 12306, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 41, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 41, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 41, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 41, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.230 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-09-21T02:18:54.231 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x5602a2f13450": { 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 6, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 2004, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 6, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 5, 2024-09-21T02:18:54.232 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 2004, 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "trackedop": { 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 105 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:54.233 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-21T02:18:54.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:18:54.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:18:54.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:54 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:18:54.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:18:54.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:18:54.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:54 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:18:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T02:18:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T02:18:54.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:54 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' 2024-09-21T02:18:55.059 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-21T02:18:55.059 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-21T02:18:55.060 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-09-21T02:18:55.060 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=20 2024-09-21T02:18:55.060 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-09-21T02:18:55.060 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:18:55.060 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:18:55.060 DEBUG:tasks.check_counter:Getting stats from h 2024-09-21T02:18:55.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.h perf dump 2024-09-21T02:18:55.345 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 317, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 320, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 908307, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 218147, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 14, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 7, 2024-09-21T02:18:55.858 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.055032046, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.022620087, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.042461934, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000346482, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 318, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.032711304, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000102865 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 38, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000008444, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000222 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 905216, 2024-09-21T02:18:55.859 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 158640 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 94, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 309, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 31058, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 459338, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 13, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 8, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.027809419, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.015346728, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.010076262, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000523674, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 303, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.030395800, 2024-09-21T02:18:55.860 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000100316 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 161, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000035844, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000222 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 392512 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1205, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 1028, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 389514, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 446152, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 9, 2024-09-21T02:18:55.861 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 5, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.128090775, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.066991677, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.065954024, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.004362607, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1032, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.083223253, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000080642 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2122, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000509930, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000240 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.862 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 0 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 28, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001976185, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000070578 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.863 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000022743, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000011371 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 2, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 2, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001680839, 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000840419 2024-09-21T02:18:55.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 11, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 16, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 16, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 13, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 8, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 2, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 2, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 3, 2024-09-21T02:18:55.865 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 38, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 32, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 6, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:55.866 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 4, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 234, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 2, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:55.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 2, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 0 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 20, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 13, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 14, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 0, 2024-09-21T02:18:55.868 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 11, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 105, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_started": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:55.869 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_write_failed": 0 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 6, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 6, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 2, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 2, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 2, 2024-09-21T02:18:55.870 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 4196308, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 0, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 5, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.013148654, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.002629730 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 0 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 20, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 24, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 4, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 19, 2024-09-21T02:18:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 33, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 14, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 16, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 24, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 8, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 2, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 2, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 0, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 119544, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 325853184 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 5, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 2, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 23, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 1, 2024-09-21T02:18:55.872 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.001680839, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000840419 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.873 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.874 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.875 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.876 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.877 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:55.878 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.879 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.880 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-09-21T02:18:55.882 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 3, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 1, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 1, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 1, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 680, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:55.884 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:55.884 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:55.884 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:55.885 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:55.885 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:55.885 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:55.885 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:55.885 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:55.886 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 262937, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 95, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:55.887 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 195, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 83201, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 368, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.888 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 42, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 12461, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 27, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 42, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 42, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.261993672, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.006237944 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 42, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 65 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.889 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 42, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 42, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 10, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 5, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 7, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 6, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-09-21T02:18:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 20, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-09-21T02:18:55.891 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 98, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 14, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 10, 2024-09-21T02:18:55.892 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 10, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 16, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 1 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 1, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:55.893 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 1612, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 1165558, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 1612, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 1612, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1165558, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 42, 2024-09-21T02:18:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 12461, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 42, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 12461, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:55.896 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 42, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 42, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 42, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 42, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:55.897 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x55595b5ab380": { 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 6, 2024-09-21T02:18:55.898 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 2004, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 6, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 5, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 2004, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "trackedop": { 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 105 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:55.899 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-21T02:18:56.020 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:55 smithi123 ceph-mon[33453]: pgmap v717: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:18:56.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:55 smithi016 ceph-mon[31245]: pgmap v717: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:18:56.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:55 smithi161 ceph-mon[34061]: pgmap v717: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-21T02:18:56.576 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=20 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:18:56.577 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:18:56.577 DEBUG:tasks.check_counter:Getting stats from k 2024-09-21T02:18:56.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.k perf dump 2024-09-21T02:18:56.826 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.b/config 2024-09-21T02:18:57.434 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1083, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 1077, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 1154016, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 411664, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 5, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 1, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.155944395, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.074516272, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.056499761, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.025870915, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1076, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.066215076, 2024-09-21T02:18:57.435 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000061538 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 879840, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 152176 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 4, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 187, 2024-09-21T02:18:57.436 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 876, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 483546, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 5, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 3, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.014123782, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.010549656, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.001847005, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.000059181, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 185, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.010295623, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000055652 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.437 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 482848 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_messages": 1552, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages": 1552, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_bytes": 399457, 2024-09-21T02:18:57.438 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_bytes": 492719, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_created_connections": 4, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_active_connections": 3, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_total_time": 0.115379096, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_send_time": 0.042738532, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_recv_time": 0.027817044, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_running_fast_dispatch_time": 0.039082323, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 1551, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.056641132, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000036519 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.439 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "cct": { 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "total_workers": 1, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "unhealthy_workers": 0 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-MDSRank": { 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 3963, 2024-09-21T02:18:57.440 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.104901572, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000026470 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "queue_len": 0, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "complete_latency": { 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "request": 0, 2024-09-21T02:18:57.441 INFO:teuthology.orchestra.run.smithi123.stdout: "reply": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "reply_latency": { 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_reply": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "forward": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_commit": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_split": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_merge": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes": 0, 2024-09-21T02:18:57.442 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_top": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_bottom": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_pinned": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_expired": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "inodes_with_caps": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "subtrees": 1, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_hit": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_forward": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_discover": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:57.443 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "traverse_lock": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "load_cent": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "q": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "exported": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "exported_inodes": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "imported": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "imported_inodes": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rfiles": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rbytes": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "root_rsnaps": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:57.444 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:57.445 INFO:teuthology.orchestra.run.smithi123.stdout: "process_request_cap_release": 0 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache": { 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_update_receipt": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_try_discover": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_send_discover": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_created": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_enqueued": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:57.446 INFO:teuthology.orchestra.run.smithi123.stdout: "strays_migrated": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_started": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_completed": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_path": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_quiesce_inode": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_flush": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_started": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:57.447 INFO:teuthology.orchestra.run.smithi123.stdout: "uninline_write_failed": 0 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log": { 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evlrg": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evadd": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evex": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evtrm": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "ev": 6, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evexg": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "evexd": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segadd": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segex": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segtrm": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "seg": 2, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segmjr": 2, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segexg": 0, 2024-09-21T02:18:57.448 INFO:teuthology.orchestra.run.smithi123.stdout: "segexd": 0, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "expos": 4194304, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "wrpos": 4196308, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rdpos": 4195917, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "jlat": { 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "replayed": 6 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mem": { 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "ino": 3, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "ino+": 3, 2024-09-21T02:18:57.449 INFO:teuthology.orchestra.run.smithi123.stdout: "ino-": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dir": 1, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dir+": 1, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dir-": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dn": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dn+": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dn-": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "cap": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "cap+": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "cap-": 0, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rss": 116844, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "heap": 316416000 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_server": { 2024-09-21T02:18:57.450 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_client_request": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_request": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_client_session": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "handle_peer_request": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "req_create_latency": { 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getattr_latency": { 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:57.451 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "req_link_latency": { 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookup_latency": { 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.452 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupino_latency": { 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupname_latency": { 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:57.453 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "req_lssnap_latency": { 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.454 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mkdir_latency": { 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mknod_latency": { 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.455 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "req_mksnap_latency": { 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "req_open_latency": { 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "req_readdir_latency": { 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.456 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rename_latency": { 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.457 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmdir_latency": { 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.458 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setattr_latency": { 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:57.459 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setlayout_latency": { 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "req_setxattr_latency": { 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "req_symlink_latency": { 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.460 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "req_unlink_latency": { 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.461 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessions": { 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "session_count": 2, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "session_add": 3, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "session_remove": 1, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_open": 2, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "sessions_stale": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "total_load": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "average_load": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "avg_session_uptime": 1801, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool": { 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:57.462 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:57.463 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_bytes": 181320, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_anon_items": 47, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:57.465 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_items": 195, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "pgmap_items": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_bytes": 8672, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_co_items": 13, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_1_items": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "unittest_2_items": 0 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter": { 2024-09-21T02:18:57.466 INFO:teuthology.orchestra.run.smithi123.stdout: "op_active": 0, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_laggy": 0, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send": 2366, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_send_bytes": 21, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_resend": 3, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_reply": 2366, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_latency": { 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2366, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 1.872954771, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000791612 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "op_inflight": 0, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "oplen_avg": { 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 2366, 2024-09-21T02:18:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 2367 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "op": 2366, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "op_r": 2366, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "op_w": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "op_rmw": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "op_pg": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_stat": 1540, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_create": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_read": 824, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_write": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writefull": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_writesame": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_append": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_zero": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_truncate": 0, 2024-09-21T02:18:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_delete": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_mapext": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_getxattr": 1, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_setxattr": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_call": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_watch": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_notify": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:57.469 INFO:teuthology.orchestra.run.smithi123.stdout: "osdop_other": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_active": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_send": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_resend": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "linger_ping": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_active": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_send": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolop_resend": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_active": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_send": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "poolstat_resend": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_active": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_send": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "statfs_resend": 0, 2024-09-21T02:18:57.470 INFO:teuthology.orchestra.run.smithi123.stdout: "command_active": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "command_send": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "command_resend": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "map_epoch": 98, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "map_full": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "map_inc": 14, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_sessions": 4, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_open": 4, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_session_close": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_laggy": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_wr": 0, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_rd": 2, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_del": 0 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "oft": { 2024-09-21T02:18:57.471 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_objs": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_updates": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "omap_total_removes": 0 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "purge_queue": { 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_executed": 0, 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:57.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 2635, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 1300536, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 2635, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 2635, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 1300536, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.473 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 104857600, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 2366, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 2009, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 2366, 2024-09-21T02:18:57.474 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 2009, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 1024, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:57.475 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 2366, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 2366, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 2366, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 2366, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:57.476 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "throttle-write_buf_throttle-0x56230cc1d520": { 2024-09-21T02:18:57.477 INFO:teuthology.orchestra.run.smithi123.stdout: "val": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "max": 3758096384, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "get_started": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "get": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "get_sum": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "take": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "take_sum": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "put": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "put_sum": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "wait": { 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "avgcount": 0, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "sum": 0.000000000, 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: "avgtime": 0.000000000 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-21T02:18:57.479 INFO:teuthology.orchestra.run.smithi123.stdout: "trackedop": { 2024-09-21T02:18:57.479 INFO:teuthology.orchestra.run.smithi123.stdout: "slow_ops_count": 0 2024-09-21T02:18:57.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-21T02:18:57.479 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-21T02:18:58.134 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:57 smithi123 ceph-mon[33453]: pgmap v718: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:18:58.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:57 smithi016 ceph-mon[31245]: pgmap v718: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:18:58.162 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:18:58.163 DEBUG:tasks.check_counter:Getting stats from c 2024-09-21T02:18:58.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.c perf dump 2024-09-21T02:18:58.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:57 smithi161 ceph-mon[34061]: pgmap v718: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-21T02:18:58.590 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 37, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 59, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 8589, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 67714, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 15, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 8, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.010314253, 2024-09-21T02:18:59.414 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.002010468, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.005512653, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000654036, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 53, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.021041828, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000397015 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 38, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000008057, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000212 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 5872 2024-09-21T02:18:59.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 83, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 281, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 28393, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 455225, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 13, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 7, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.025823652, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.015405532, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.008602068, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000304808, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 277, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.024505871, 2024-09-21T02:18:59.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000088468 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 155, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000059237, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000382 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1744, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 393392 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1491, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1297, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1274616, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 594048, 2024-09-21T02:18:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 8, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.154219320, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.071161357, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.098001262, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.004846614, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1303, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.084856665, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000065124 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2133, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000480791, 2024-09-21T02:18:59.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000225 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 889360, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 155008 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 51, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.002019133, 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000039590 2024-09-21T02:18:59.419 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000022361, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000011180 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 2, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 1, 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-09-21T02:18:59.420 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000411522, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000411522 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 1, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 11, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 16, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-09-21T02:18:59.421 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 16, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 13, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 8, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 3, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 38, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 30, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 7, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 1000, 2024-09-21T02:18:59.422 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 1, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 4, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 234, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:18:59.423 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 1 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.424 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 18, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 12, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 14, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:18:59.425 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 11, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 105, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_started": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_succeeded": 0, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_write_failed": 0 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.426 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 6, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 6, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 2, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-09-21T02:18:59.427 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4196308, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 0, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 5, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.010071115, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.002014223 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 0 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 20, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 24, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 4, 2024-09-21T02:18:59.428 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 19, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 32, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 13, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 16, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 24, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 8, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 1, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 1, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 119624, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 325857280 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 3, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 2, 2024-09-21T02:18:59.429 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 23, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000411522, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000411522 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.430 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.434 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.435 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.436 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.437 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.438 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:18:59.439 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 3, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 1, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 2, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 1, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 1, 2024-09-21T02:18:59.440 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 682, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:18:59.441 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-09-21T02:18:59.442 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 254889, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 92, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-09-21T02:18:59.443 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9640, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 195, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 82529, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 360, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-09-21T02:18:59.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 41, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 12306, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 33, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 41, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 41, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.230994917, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.005634022 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 41, 2024-09-21T02:18:59.445 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 63 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 41, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 41, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 10, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 5, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 7, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-09-21T02:18:59.446 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 6, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 20, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:18:59.447 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-09-21T02:18:59.448 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 98, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 14, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 10, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 10, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 14, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 0, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 1 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.449 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 1, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.450 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 1607, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1148530, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 1607, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 1607, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1148530, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.451 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 41, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 12306, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 41, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 12306, 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:18:59.452 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 41, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 41, 2024-09-21T02:18:59.453 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 41, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 41, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:18:59.454 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.455 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55acc39911e0": { 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 6, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 2004, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 6, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 5, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2004, 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:18:59.456 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: "trackedop": { 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 105 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:18:59.457 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-21T02:19:00.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:18:59 smithi016 ceph-mon[31245]: pgmap v719: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:19:00.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:18:59 smithi161 ceph-mon[34061]: pgmap v719: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:19:00.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:18:59 smithi123 ceph-mon[33453]: pgmap v719: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-21T02:19:00.376 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-21T02:19:00.376 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-21T02:19:00.376 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-09-21T02:19:00.376 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=18 2024-09-21T02:19:00.376 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-09-21T02:19:00.377 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:19:00.377 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:19:00.377 DEBUG:tasks.check_counter:Getting stats from f 2024-09-21T02:19:00.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.f perf dump 2024-09-21T02:19:00.637 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1760, 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1760, 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 4439563, 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 558866, 2024-09-21T02:19:01.392 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 2, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.210432318, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.084179873, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.050812220, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.060914953, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1759, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.084152455, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000047841 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.393 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 268, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 262, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 879880, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 151898, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.046244915, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.021494259, 2024-09-21T02:19:01.394 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.033610794, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000100850, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 261, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.021166333, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000081097 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 878224, 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 150560 2024-09-21T02:19:01.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 821, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1002, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 201951, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 652905, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.106278292, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.051734521, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.017591707, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.028571971, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1000, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.045624102, 2024-09-21T02:19:01.396 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000045624 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 392144 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.397 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 4239, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.123388847, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000029108 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.398 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-09-21T02:19:01.399 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 169, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 169, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 169, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 2, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-09-21T02:19:01.400 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 4, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 234, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 6, 2024-09-21T02:19:01.401 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-09-21T02:19:01.402 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-09-21T02:19:01.403 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-09-21T02:19:01.404 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 0, 2024-09-21T02:19:01.405 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_started": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_succeeded": 0, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_write_failed": 0 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-09-21T02:19:01.406 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 532, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-09-21T02:19:01.407 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 8111512, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 8107266, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 532 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 126, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 144, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 18, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 20, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 21, 2024-09-21T02:19:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 1, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 169, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 169, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 119760, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 317468672 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-09-21T02:19:01.409 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-09-21T02:19:01.410 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.411 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-09-21T02:19:01.412 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-09-21T02:19:01.413 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-09-21T02:19:01.414 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.415 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-09-21T02:19:01.416 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-09-21T02:19:01.417 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-09-21T02:19:01.418 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.419 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 4, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 2, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 2, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-09-21T02:19:01.420 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 1821, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:19:01.421 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:19:01.422 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 189423, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 51, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-09-21T02:19:01.423 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9640, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 195, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 483018, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 2253, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-09-21T02:19:01.424 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 2580, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 26, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 4, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 2580, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2580, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 1.519966698, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000589134 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.425 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2580, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 2581 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 2580, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 2580, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 1638, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 939, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-09-21T02:19:01.426 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 2, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-09-21T02:19:01.427 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-09-21T02:19:01.428 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 98, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 14, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 5, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 5, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-09-21T02:19:01.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-09-21T02:19:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2845, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 5248873, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2845, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2845, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 5248873, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.431 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2580, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 3917218, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2580, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 3917218, 2024-09-21T02:19:01.432 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2580, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2580, 2024-09-21T02:19:01.433 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2580, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2580, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:01.434 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x565125643380": { 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:01.435 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "trackedop": { 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:01.436 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-21T02:19:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:01 smithi016 ceph-mon[31245]: pgmap v720: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:19:02.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:01 smithi161 ceph-mon[34061]: pgmap v720: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:19:02.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:01 smithi123 ceph-mon[33453]: pgmap v720: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-21T02:19:02.283 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-21T02:19:02.283 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-21T02:19:02.283 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-09-21T02:19:02.283 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-09-21T02:19:02.283 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-09-21T02:19:02.284 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:19:02.284 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:19:02.284 DEBUG:tasks.check_counter:Getting stats from i 2024-09-21T02:19:02.284 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.i perf dump 2024-09-21T02:19:02.533 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 431, 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 456, 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 948237, 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 244612, 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 13, 2024-09-21T02:19:03.080 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.063064099, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.028033223, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.043496152, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000220209, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 455, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.031972118, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000070268 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 261, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000061744, 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000236 2024-09-21T02:19:03.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 898192, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 154496 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1160, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1161, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 372946, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 713827, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 9, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.122500589, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.061041348, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.061921422, 2024-09-21T02:19:03.082 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.005841890, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1161, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.072299628, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000062273 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2042, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000478905, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000234 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-21T02:19:03.083 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 393616 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 20, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 22, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 6804, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 56199, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 12, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 8, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.006977452, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.000575445, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.005814934, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000320512, 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:03.084 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 19, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.018839168, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000991535 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 4, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000373, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000093 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-09-21T02:19:03.085 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 49, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.001452792, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000029648 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:03.086 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000024684, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000012342 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 1, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 1, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000364155, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000364155 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.087 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 11, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 16, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 16, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 13, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 8, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 1, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 1, 2024-09-21T02:19:03.088 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 3, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 37, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 30, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 7, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 1000, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 1, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-09-21T02:19:03.089 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 4, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 234, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:19:03.090 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 1, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 1, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-09-21T02:19:03.091 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 22, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 14, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 14, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-09-21T02:19:03.092 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 11, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 105, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_started": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_succeeded": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_write_failed": 0 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-09-21T02:19:03.093 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 5, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 5, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 2, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4195917, 2024-09-21T02:19:03.094 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 0, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 4, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.009389439, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.002347359 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 0 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 20, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 24, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 4, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 19, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 33, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 14, 2024-09-21T02:19:03.095 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 16, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 24, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 8, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 1, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 1, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 119952, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 325857280 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 1, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-09-21T02:19:03.096 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 1, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 14, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000364155, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000364155 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.097 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.099 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-09-21T02:19:03.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.101 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.102 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 1, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 2, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 1, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 1, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 1, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 1, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 714, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-09-21T02:19:03.108 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-09-21T02:19:03.109 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-09-21T02:19:03.110 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 262519, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 90, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:19:03.111 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9640, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 195, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 82833, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 360, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-09-21T02:19:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 40, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 11919, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 28, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 40, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 40, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.215995284, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.005399882 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-09-21T02:19:03.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 40, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 63 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 40, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 40, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 10, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 4, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 6, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-09-21T02:19:03.114 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 6, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 20, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-09-21T02:19:03.115 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-09-21T02:19:03.116 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 98, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 14, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 9, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 9, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 16, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 0, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 1 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-09-21T02:19:03.117 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 1, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.118 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 1607, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1165438, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 1607, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 1607, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1165438, 2024-09-21T02:19:03.119 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 40, 2024-09-21T02:19:03.120 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 11919, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 40, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 11919, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:03.121 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 40, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 40, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 40, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 40, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.122 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.123 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55ef35697450": { 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 5, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1613, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 5, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:03.124 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:03.125 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 4, 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1613, 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "trackedop": { 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 105 2024-09-21T02:19:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:03.127 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-21T02:19:03.902 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=22 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:19:03.903 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:19:03.903 DEBUG:tasks.check_counter:Getting stats from l 2024-09-21T02:19:03.903 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph daemon mds.l perf dump 2024-09-21T02:19:03.927 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:03 smithi161 ceph-mon[34061]: pgmap v721: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:04.151 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.c/config 2024-09-21T02:19:04.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:03 smithi016 ceph-mon[31245]: pgmap v721: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:04.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:03 smithi123 ceph-mon[33453]: pgmap v721: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 831, 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 831, 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 221218, 2024-09-21T02:19:04.687 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 263497, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 2, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.088112333, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.037943285, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.020853387, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.022030175, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 830, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.039320275, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000047373 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 448 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 271, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 265, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 881486, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 154098, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.045861716, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.020358334, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.036039488, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000059838, 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:04.689 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 264, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.021703937, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000082211 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 880384, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 153456 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.690 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1550, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1735, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 454928, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 977128, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 4, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.205375230, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.085878601, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.044294561, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.060605530, 2024-09-21T02:19:04.691 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1733, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.081846140, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000047228 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 484864 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-09-21T02:19:04.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3952, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.123685203, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000031296 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.693 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-09-21T02:19:04.694 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 1, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-09-21T02:19:04.695 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 0, 2024-09-21T02:19:04.696 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-21T02:19:04.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-09-21T02:19:04.698 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_path": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_quiesce_inode": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_started": 0, 2024-09-21T02:19:04.699 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_succeeded": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "uninline_write_failed": 0 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 5, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-09-21T02:19:04.700 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4195917, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 4195424, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 5 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 3, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 3, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 0, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 1, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 1, 2024-09-21T02:19:04.701 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 118736, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 316424192 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-09-21T02:19:04.702 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.703 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-09-21T02:19:04.704 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.705 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.706 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.707 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-09-21T02:19:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.709 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.711 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-09-21T02:19:04.712 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 1, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 2, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 1, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 1, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 1826, 2024-09-21T02:19:04.713 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-09-21T02:19:04.714 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-09-21T02:19:04.715 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 198231, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 55, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-09-21T02:19:04.716 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9640, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 195, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 8672, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 13, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-09-21T02:19:04.717 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 2377, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 21, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 3, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 2377, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2377, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 4.060911156, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.001708418 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.718 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2377, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 2378 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 2377, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 2377, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 1548, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 827, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-09-21T02:19:04.719 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 1, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-09-21T02:19:04.720 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-09-21T02:19:04.721 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 98, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 14, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 6, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 6, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-09-21T02:19:04.722 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-09-21T02:19:04.723 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2648, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1302545, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2648, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2648, 2024-09-21T02:19:04.724 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1302545, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:04.725 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2377, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 1618, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2377, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1618, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:04.726 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2377, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2377, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2377, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2377, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.727 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:19:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55aa8007b520": { 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-09-21T02:19:04.729 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "trackedop": { 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-21T02:19:04.730 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-09-21T02:19:05.427 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-09-21T02:19:05.428 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-09-21T02:19:05.428 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-21T02:19:05.438 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-21T02:19:05.438 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:19:05.438 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:19:05.472 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:19:05.472 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:19:05.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:19:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:05.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:05 smithi016 ceph-mon[31245]: pgmap v722: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T02:19:06.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:05 smithi161 ceph-mon[34061]: pgmap v722: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T02:19:06.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:05 smithi123 ceph-mon[33453]: pgmap v722: 129 pgs: 129 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-09-21T02:19:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.509+0000 7f2333b6a640 1 -- 172.21.15.16:0/2085753071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c1027a0 msgr2=0x7f232c102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.509+0000 7f2333b6a640 1 --2- 172.21.15.16:0/2085753071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c1027a0 0x7f232c102bb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f231c009cd0 tx=0x7f231c02a930 comp rx=0 tx=0).stop 2024-09-21T02:19:06.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.511+0000 7f2333b6a640 1 -- 172.21.15.16:0/2085753071 shutdown_connections 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.511+0000 7f2333b6a640 1 --2- 172.21.15.16:0/2085753071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c107be0 0x7f232c1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.511+0000 7f2333b6a640 1 --2- 172.21.15.16:0/2085753071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1030f0 0x7f232c107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.511+0000 7f2333b6a640 1 --2- 172.21.15.16:0/2085753071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c1027a0 0x7f232c102bb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.511+0000 7f2333b6a640 1 -- 172.21.15.16:0/2085753071 >> 172.21.15.16:0/2085753071 conn(0x7f232c079730 msgr2=0x7f232c079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.512+0000 7f2333b6a640 1 -- 172.21.15.16:0/2085753071 shutdown_connections 2024-09-21T02:19:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.512+0000 7f2333b6a640 1 -- 172.21.15.16:0/2085753071 wait complete. 2024-09-21T02:19:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.513+0000 7f2333b6a640 1 Processor -- start 2024-09-21T02:19:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.513+0000 7f2333b6a640 1 -- start start 2024-09-21T02:19:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1027a0 0x7f232c116db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c1030f0 0x7f232c110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f232c11a080 con 0x7f232c1027a0 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f232c119f00 con 0x7f232c107be0 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2333b6a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f232c11a200 con 0x7f232c1030f0 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2332107640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.514+0000 7f2332107640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.16:59360/0 (socket says 172.21.15.16:59360) 2024-09-21T02:19:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.515+0000 7f2332107640 1 -- 172.21.15.16:0/4220705268 learned_addr learned my addr 172.21.15.16:0/4220705268 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.515+0000 7f2331906640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1027a0 0x7f232c116db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.517+0000 7f2331105640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c1030f0 0x7f232c110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.518+0000 7f2332107640 1 -- 172.21.15.16:0/4220705268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c1030f0 msgr2=0x7f232c110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.518+0000 7f2332107640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c1030f0 0x7f232c110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.518+0000 7f2332107640 1 -- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1027a0 msgr2=0x7f232c116db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.518+0000 7f2332107640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1027a0 0x7f232c116db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.518+0000 7f2332107640 1 -- 172.21.15.16:0/4220705268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c115a20 con 0x7f232c107be0 2024-09-21T02:19:06.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.526+0000 7f2332107640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f232400b3f0 tx=0x7f232400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.528+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324015020 con 0x7f232c107be0 2024-09-21T02:19:06.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.528+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2324011cc0 con 0x7f232c107be0 2024-09-21T02:19:06.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.529+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324010d00 con 0x7f232c107be0 2024-09-21T02:19:06.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.532+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c115d10 con 0x7f232c107be0 2024-09-21T02:19:06.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.532+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c116250 con 0x7f232c107be0 2024-09-21T02:19:06.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.535+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f232c06b560 con 0x7f232c107be0 2024-09-21T02:19:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.542+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f2324018020 con 0x7f232c107be0 2024-09-21T02:19:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.543+0000 7f232affd640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 0x7f23140b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.543+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9570+0+0 (secure 0 0 0) 0x7f23240e7280 con 0x7f232c107be0 2024-09-21T02:19:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.544+0000 7f2331906640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 0x7f23140b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.545+0000 7f2331906640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 0x7f23140b8d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f231c006010 tx=0x7f231c000f40 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.550+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f232401f020 con 0x7f232c107be0 2024-09-21T02:19:06.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.948+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f232c1076a0 con 0x7f232c107be0 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.950+0000 7f232affd640 1 -- 172.21.15.16:0/4220705268 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v99) ==== 81+0+888 (secure 0 0 0) 0x7f23240ad350 con 0x7f232c107be0 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3095458517 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2441291809 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3677616942 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4075907232 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2368126584 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:06.952 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3423016963 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3586667721 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429917716 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3010596393 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-09-21T02:19:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.954+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 msgr2=0x7f23140b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.954+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 0x7f23140b8d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f231c006010 tx=0x7f231c000f40 comp rx=0 tx=0).stop 2024-09-21T02:19:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.954+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 msgr2=0x7f232c110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.954+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f232400b3f0 tx=0x7f232400b8b0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 shutdown_connections 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f23140b6880 0x7f23140b8d90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f232c107be0 0x7f232c110710 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f232c1030f0 0x7f232c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 --2- 172.21.15.16:0/4220705268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f232c1027a0 0x7f232c116db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.955+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 >> 172.21.15.16:0/4220705268 conn(0x7f232c079730 msgr2=0x7f232c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.956+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 shutdown_connections 2024-09-21T02:19:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:06.956+0000 7f2333b6a640 1 -- 172.21.15.16:0/4220705268 wait complete. 2024-09-21T02:19:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:06 smithi016 ceph-mon[31245]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T02:19:07.164 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:06 smithi161 ceph-mon[34061]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T02:19:07.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:06 smithi123 ceph-mon[33453]: osdmap e99: 12 total, 12 up, 12 in 2024-09-21T02:19:07.739 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-21T02:19:07.739 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T02:19:07.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T02:19:08.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:07 smithi016 ceph-mon[31245]: pgmap v724: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:08.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:07 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/4220705268' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:08.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:07 smithi161 ceph-mon[34061]: pgmap v724: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:08.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:07 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/4220705268' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:07 smithi123 ceph-mon[33453]: pgmap v724: 129 pgs: 3 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 114 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:08.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:07 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/4220705268' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.396+0000 7f31afed3640 1 -- 172.21.15.16:0/32124860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a810bff0 msgr2=0x7f31a810c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.396+0000 7f31afed3640 1 --2- 172.21.15.16:0/32124860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a810bff0 0x7f31a810c400 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009900 tx=0x7f31a402a440 comp rx=0 tx=0).stop 2024-09-21T02:19:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.400+0000 7f31afed3640 1 -- 172.21.15.16:0/32124860 shutdown_connections 2024-09-21T02:19:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.400+0000 7f31afed3640 1 --2- 172.21.15.16:0/32124860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a8117b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.400+0000 7f31afed3640 1 --2- 172.21.15.16:0/32124860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a8112dc0 0x7f31a81151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.400+0000 7f31afed3640 1 --2- 172.21.15.16:0/32124860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a810bff0 0x7f31a810c400 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.400+0000 7f31afed3640 1 -- 172.21.15.16:0/32124860 >> 172.21.15.16:0/32124860 conn(0x7f31a80712f0 msgr2=0x7f31a8073e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.403+0000 7f31afed3640 1 -- 172.21.15.16:0/32124860 shutdown_connections 2024-09-21T02:19:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.406+0000 7f31afed3640 1 -- 172.21.15.16:0/32124860 wait complete. 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 Processor -- start 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 -- start start 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 0x7f31a81a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a8112dc0 0x7f31a81a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a81a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31a811b120 con 0x7f31a8115720 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31a811afa0 con 0x7f31a8112dc0 2024-09-21T02:19:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.407+0000 7f31afed3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31a811b2a0 con 0x7f31a810bff0 2024-09-21T02:19:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.411+0000 7f31ae470640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a81a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.411+0000 7f31ae470640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a81a7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52828/0 (socket says 172.21.15.16:52828) 2024-09-21T02:19:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.411+0000 7f31ae470640 1 -- 172.21.15.16:0/3306099010 learned_addr learned my addr 172.21.15.16:0/3306099010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.411+0000 7f31adc6f640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 0x7f31a81a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.413+0000 7f31adc6f640 1 -- 172.21.15.16:0/3306099010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a8112dc0 msgr2=0x7f31a81a20f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-21T02:19:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.413+0000 7f31adc6f640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a8112dc0 0x7f31a81a20f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.413+0000 7f31adc6f640 1 -- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 msgr2=0x7f31a81a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.413+0000 7f31adc6f640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a81a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.413+0000 7f31adc6f640 1 -- 172.21.15.16:0/3306099010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a81a7ac0 con 0x7f31a810bff0 2024-09-21T02:19:09.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.416+0000 7f31adc6f640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 0x7f31a81a1b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31a402aa30 tx=0x7f31a4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.422+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4047070 con 0x7f31a810bff0 2024-09-21T02:19:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.422+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31a4031400 con 0x7f31a810bff0 2024-09-21T02:19:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.422+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a81a7d50 con 0x7f31a810bff0 2024-09-21T02:19:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.422+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a810ac50 con 0x7f31a810bff0 2024-09-21T02:19:09.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.424+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4035030 con 0x7f31a810bff0 2024-09-21T02:19:09.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.429+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f318c0052c0 con 0x7f31a810bff0 2024-09-21T02:19:09.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.433+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f31a404a070 con 0x7f31a810bff0 2024-09-21T02:19:09.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.435+0000 7f319effd640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 0x7f31940b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.435+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9570+0+0 (secure 0 0 0) 0x7f31a4111510 con 0x7f31a810bff0 2024-09-21T02:19:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.435+0000 7f31ad46e640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 0x7f31940b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.436+0000 7f31ad46e640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 0x7f31940b8d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31a8114af0 tx=0x7f3198006bd0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.440+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f31a40d75e0 con 0x7f31a810bff0 2024-09-21T02:19:09.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.840+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f318c005880 con 0x7f31a810bff0 2024-09-21T02:19:09.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.843+0000 7f319effd640 1 -- 172.21.15.16:0/3306099010 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v99) ==== 81+0+888 (secure 0 0 0) 0x7f31a40db9c0 con 0x7f31a810bff0 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3095458517 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2441291809 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3677616942 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4075907232 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2368126584 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1610958069 2024-09-22T01:56:22.535497+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3423016963 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3586667721 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429917716 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/4160624590 2024-09-22T01:57:51.923009+0000 2024-09-21T02:19:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3010596393 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:09.846 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1946001099 2024-09-22T01:56:49.870093+0000 2024-09-21T02:19:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:listed 15 entries 2024-09-21T02:19:09.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.848+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 msgr2=0x7f31940b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:09.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.848+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 0x7f31940b8d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31a8114af0 tx=0x7f3198006bd0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.848+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 msgr2=0x7f31a81a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:09.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.848+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 0x7f31a81a1b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31a402aa30 tx=0x7f31a4003b90 comp rx=0 tx=0).stop 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.849+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 shutdown_connections 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.850+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f31940b6880 0x7f31940b8d90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.850+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a8115720 0x7f31a81a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.850+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a8112dc0 0x7f31a81a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.850+0000 7f31afed3640 1 --2- 172.21.15.16:0/3306099010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31a810bff0 0x7f31a81a1b50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:09.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.850+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 >> 172.21.15.16:0/3306099010 conn(0x7f31a80712f0 msgr2=0x7f31a8115b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.854+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 shutdown_connections 2024-09-21T02:19:09.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:09.854+0000 7f31afed3640 1 -- 172.21.15.16:0/3306099010 wait complete. 2024-09-21T02:19:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:09 smithi016 ceph-mon[31245]: pgmap v725: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:09 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3306099010' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:10.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:09 smithi161 ceph-mon[34061]: pgmap v725: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:10.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:09 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3306099010' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:09 smithi123 ceph-mon[33453]: pgmap v725: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 17 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:10.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:09 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3306099010' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T02:19:10.644 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-09-21T02:19:10.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:19:10.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-21T02:19:10.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[87558]: fuse finished with error 0 and tester_r 0 2024-09-21T02:19:10.720 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T02:19:12.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:11 smithi016 ceph-mon[31245]: pgmap v726: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:12.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:11 smithi161 ceph-mon[34061]: pgmap v726: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:12.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:11 smithi123 ceph-mon[33453]: pgmap v726: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-21T02:19:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:13 smithi016 ceph-mon[31245]: pgmap v727: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T02:19:14.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:13 smithi161 ceph-mon[34061]: pgmap v727: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T02:19:14.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:13 smithi123 ceph-mon[33453]: pgmap v727: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 16 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-21T02:19:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:15 smithi016 ceph-mon[31245]: pgmap v728: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:16.165 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:15 smithi161 ceph-mon[34061]: pgmap v728: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:16.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:15 smithi123 ceph-mon[33453]: pgmap v728: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:16.821 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:19:16.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:19:16.822 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-21T02:19:16.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:19:16.850 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-09-21T02:19:16.920 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-21T02:19:16.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:19:16.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T02:19:16.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T02:19:16.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-09-21T02:19:17.096 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T02:19:17.107 INFO:tasks.cephadm:Teardown begin 2024-09-21T02:19:17.107 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-21T02:19:17.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:19:17.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:17 smithi123 ceph-mon[33453]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T02:19:17.375 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:17.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:17 smithi016 ceph-mon[31245]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T02:19:17.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:17 smithi161 ceph-mon[34061]: osdmap e100: 12 total, 12 up, 12 in 2024-09-21T02:19:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.887+0000 7fbedcb8b640 1 -- 172.21.15.16:0/1320823460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 msgr2=0x7fbed810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.887+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/1320823460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed810f520 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009ca0 tx=0x7fbed002a8c0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.887+0000 7fbedcb8b640 1 -- 172.21.15.16:0/1320823460 shutdown_connections 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.887+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/1320823460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed810f520 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/1320823460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 0x7fbed81087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/1320823460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 0x7fbed8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 -- 172.21.15.16:0/1320823460 >> 172.21.15.16:0/1320823460 conn(0x7fbed8079730 msgr2=0x7fbed8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 -- 172.21.15.16:0/1320823460 shutdown_connections 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 -- 172.21.15.16:0/1320823460 wait complete. 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 Processor -- start 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.888+0000 7fbedcb8b640 1 -- start start 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 0x7fbed80ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 0x7fbed80ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbed8112860 con 0x7fbed8108cf0 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbed81126e0 con 0x7fbed81083a0 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbedcb8b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbed81129e0 con 0x7fbed8102a60 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51916/0 (socket says 172.21.15.16:51916) 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 -- 172.21.15.16:0/3794219910 learned_addr learned my addr 172.21.15.16:0/3794219910 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed5d9b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 0x7fbed80ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed659c640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 0x7fbed80ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 -- 172.21.15.16:0/3794219910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 msgr2=0x7fbed80ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 0x7fbed80ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 -- 172.21.15.16:0/3794219910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 msgr2=0x7fbed80ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 0x7fbed80ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.889+0000 7fbed6d9d640 1 -- 172.21.15.16:0/3794219910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed81a85b0 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbed6d9d640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005cc0 tx=0x7fbed0032a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0032e00 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed81a8840 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbed00044d0 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0030b70 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.890+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed81a8d80 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.891+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed8102e70 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.893+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7fbed0030d10 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.894+0000 7fbecf7fe640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 0x7fbec40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.894+0000 7fbed659c640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 0x7fbec40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.894+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9482+0+0 (secure 0 0 0) 0x7fbed0108fa0 con 0x7fbed8108cf0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.894+0000 7fbed659c640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 0x7fbec40b8b90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005fd0 tx=0x7fbec8005850 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:17.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:17.897+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbed00cf130 con 0x7fbed8108cf0 2024-09-21T02:19:18.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.123+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbed8100970 con 0x7fbed8108cf0 2024-09-21T02:19:18.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:18 smithi016 ceph-mon[31245]: pgmap v730: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:18.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.124+0000 7fbecf7fe640 1 -- 172.21.15.16:0/3794219910 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7fbed00d3510 con 0x7fbed8108cf0 2024-09-21T02:19:18.126 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:19:18.126 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:19:18.126 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:19:18.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 msgr2=0x7fbec40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 0x7fbec40b8b90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005fd0 tx=0x7fbec8005850 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 msgr2=0x7fbed81000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005cc0 tx=0x7fbed0032a10 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 shutdown_connections 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7fbec40b6680 0x7fbec40b8b90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed8108cf0 0x7fbed81000f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbed81083a0 0x7fbed80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 --2- 172.21.15.16:0/3794219910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbed8102a60 0x7fbed80ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 >> 172.21.15.16:0/3794219910 conn(0x7fbed8079730 msgr2=0x7fbed806b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 shutdown_connections 2024-09-21T02:19:18.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:18.126+0000 7fbedcb8b640 1 -- 172.21.15.16:0/3794219910 wait complete. 2024-09-21T02:19:18.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:18 smithi123 ceph-mon[33453]: pgmap v730: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:18.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:18 smithi161 ceph-mon[34061]: pgmap v730: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:18.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T02:19:19.106 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:19 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3794219910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:19.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:19 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3794219910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:19.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:19 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3794219910' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:20.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:20 smithi123 ceph-mon[33453]: pgmap v731: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:20.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:20 smithi016 ceph-mon[31245]: pgmap v731: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:20.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:20 smithi161 ceph-mon[34061]: pgmap v731: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-21T02:19:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/1392815260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa81049b0 msgr2=0x7f0aa8104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/1392815260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa81049b0 0x7f0aa8104dc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0a98009900 tx=0x7f0a9802a410 comp rx=0 tx=0).stop 2024-09-21T02:19:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/1392815260 shutdown_connections 2024-09-21T02:19:21.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/1392815260 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa810de10 0x7f0aa8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/1392815260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa8105300 0x7f0aa810d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/1392815260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa81049b0 0x7f0aa8104dc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/1392815260 >> 172.21.15.16:0/1392815260 conn(0x7f0aa8079730 msgr2=0x7f0aa8079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/1392815260 shutdown_connections 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.718+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/1392815260 wait complete. 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 Processor -- start 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 -- start start 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa81049b0 0x7f0aa819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8105300 0x7f0aa819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8113990 con 0x7f0aa810de10 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8113810 con 0x7f0aa81049b0 2024-09-21T02:19:21.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aaeb5a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8113b10 con 0x7f0aa8105300 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aad0f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aad0f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51932/0 (socket says 172.21.15.16:51932) 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aad0f7640 1 -- 172.21.15.16:0/2101957934 learned_addr learned my addr 172.21.15.16:0/2101957934 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.719+0000 7f0aac8f6640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa81049b0 0x7f0aa819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aa7fff640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8105300 0x7f0aa819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 -- 172.21.15.16:0/2101957934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8105300 msgr2=0x7f0aa819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8105300 0x7f0aa819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 -- 172.21.15.16:0/2101957934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa81049b0 msgr2=0x7f0aa819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa81049b0 0x7f0aa819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 -- 172.21.15.16:0/2101957934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa81a03b0 con 0x7f0aa810de10 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aad0f7640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000eaa0 tx=0x7f0aa000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000cdb0 con 0x7f0aa810de10 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0aa0005020 con 0x7f0aa810de10 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa81a06a0 con 0x7f0aa810de10 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa00116d0 con 0x7f0aa810de10 2024-09-21T02:19:21.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.720+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa81a6e00 con 0x7f0aa810de10 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.721+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa8104dc0 con 0x7f0aa810de10 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.723+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f0aa00051d0 con 0x7f0aa810de10 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.724+0000 7f0aa5ffb640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 0x7f0a900b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.724+0000 7f0aac8f6640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 0x7f0a900b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.724+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9482+0+0 (secure 0 0 0) 0x7f0aa00e8010 con 0x7f0aa810de10 2024-09-21T02:19:21.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.725+0000 7f0aac8f6640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 0x7f0a900b8b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0a980057c0 tx=0x7f0a98000bc0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:21.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.727+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f0aa0016030 con 0x7f0aa810de10 2024-09-21T02:19:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.955+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0aa819b570 con 0x7f0aa810de10 2024-09-21T02:19:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.956+0000 7f0aa5ffb640 1 -- 172.21.15.16:0/2101957934 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11992 (secure 0 0 0) 0x7f0aa819b570 con 0x7f0aa810de10 2024-09-21T02:19:21.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:19:21.957 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"btime":"2024-09-21T02:07:08:176803+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14781,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3026745530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3026745530},{"type":"v1","addr":"172.21.15.16:6835","nonce":3026745530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3790973256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3790973256},{"type":"v1","addr":"172.21.15.16:6837","nonce":3790973256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-21T02:04:34.359195+0000","modified":"2024-09-21T02:07:07.873572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24710,"mds_1":24683,"mds_2":24667,"mds_3":24659,"mds_4":24655},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14841":{"gid":14841,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6839/1623043260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1623043260},{"type":"v1","addr":"172.21.15.16:6839","nonce":1623043260}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14853":{"gid":14853,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6841/3323661058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3323661058},{"type":"v1","addr":"172.21.15.16:6841","nonce":3323661058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24595":{"gid":24595,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2556025136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2556025136},{"type":"v1","addr":"172.21.15.161:6833","nonce":2556025136}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24626":{"gid":24626,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6833/3641506877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3641506877},{"type":"v1","addr":"172.21.15.123:6833","nonce":3641506877}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24643":{"gid":24643,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/1279942273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":1279942273},{"type":"v1","addr":"172.21.15.161:6835","nonce":1279942273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.161:6837/2373808610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":2373808610},{"type":"v1","addr":"172.21.15.161:6837","nonce":2373808610}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24659":{"gid":24659,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":20,"addr":"172.21.15.123:6835/2421556237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2421556237},{"type":"v1","addr":"172.21.15.123:6835","nonce":2421556237}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24667":{"gid":24667,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.161:6839/2696881795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2696881795},{"type":"v1","addr":"172.21.15.161:6839","nonce":2696881795}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24683":{"gid":24683,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":17,"addr":"172.21.15.123:6837/1023560595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1023560595},{"type":"v1","addr":"172.21.15.123:6837","nonce":1023560595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24710":{"gid":24710,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6839/1116723582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":1116723582},{"type":"v1","addr":"172.21.15.123:6839","nonce":1116723582}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24710,"qdb_cluster":[24710,24683,24667,24659,24655]},"id":1}]} 2024-09-21T02:19:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 msgr2=0x7f0a900b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 0x7f0a900b8b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0a980057c0 tx=0x7f0a98000bc0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 msgr2=0x7f0aa819fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000eaa0 tx=0x7f0aa000ef60 comp rx=0 tx=0).stop 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 shutdown_connections 2024-09-21T02:19:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f0a900b6680 0x7f0a900b8b90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa810de10 0x7f0aa819fc80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8105300 0x7f0aa819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 --2- 172.21.15.16:0/2101957934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0aa81049b0 0x7f0aa819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 >> 172.21.15.16:0/2101957934 conn(0x7f0aa8079730 msgr2=0x7f0aa8100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 shutdown_connections 2024-09-21T02:19:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:21.958+0000 7f0aaeb5a640 1 -- 172.21.15.16:0/2101957934 wait complete. 2024-09-21T02:19:22.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:22 smithi123 ceph-mon[33453]: pgmap v732: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-21T02:19:22.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:22 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2101957934' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:22.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:22 smithi016 ceph-mon[31245]: pgmap v732: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-21T02:19:22.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:22 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2101957934' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:22 smithi161 ceph-mon[34061]: pgmap v732: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-21T02:19:22.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:22 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2101957934' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T02:19:22.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T02:19:22.899 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 -- 172.21.15.16:0/2770343732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 msgr2=0x7f8568102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:23.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 --2- 172.21.15.16:0/2770343732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f8568102ef0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8564009900 tx=0x7f856402a410 comp rx=0 tx=0).stop 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 -- 172.21.15.16:0/2770343732 shutdown_connections 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 --2- 172.21.15.16:0/2770343732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8568108d20 0x7f856810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 --2- 172.21.15.16:0/2770343732 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85681083d0 0x7f85681087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 --2- 172.21.15.16:0/2770343732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f8568102ef0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 -- 172.21.15.16:0/2770343732 >> 172.21.15.16:0/2770343732 conn(0x7f8568079760 msgr2=0x7f8568079b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 -- 172.21.15.16:0/2770343732 shutdown_connections 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 -- 172.21.15.16:0/2770343732 wait complete. 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.407+0000 7f8570557640 1 Processor -- start 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 -- start start 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85681083d0 0x7f856819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 0x7f856819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8568112840 con 0x7f8568102ae0 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f85681126c0 con 0x7f85681083d0 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f8570557640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f85681129c0 con 0x7f8568108d20 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51956/0 (socket says 172.21.15.16:51956) 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 -- 172.21.15.16:0/2836863647 learned_addr learned my addr 172.21.15.16:0/2836863647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856daf2640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85681083d0 0x7f856819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856eaf4640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 0x7f856819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 -- 172.21.15.16:0/2836863647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 msgr2=0x7f856819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 0x7f856819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 -- 172.21.15.16:0/2836863647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85681083d0 msgr2=0x7f856819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85681083d0 0x7f856819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856e2f3640 1 -- 172.21.15.16:0/2836863647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85681a02b0 con 0x7f8568102ae0 2024-09-21T02:19:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.408+0000 7f856eaf4640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 0x7f856819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f856e2f3640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8564009230 tx=0x7f8564004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856403f030 con 0x7f8568102ae0 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8564003a80 con 0x7f8568102ae0 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85681a0540 con 0x7f8568102ae0 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564035070 con 0x7f8568102ae0 2024-09-21T02:19:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.409+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85681a0a20 con 0x7f8568102ae0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.410+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8568103d10 con 0x7f8568102ae0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.411+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8564042030 con 0x7f8568102ae0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.412+0000 7f855f7fe640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 0x7f85540b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.412+0000 7f856daf2640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 0x7f85540b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.412+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9482+0+0 (secure 0 0 0) 0x7f8564108670 con 0x7f8568102ae0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.413+0000 7f856daf2640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 0x7f85540b8bc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8568079fb0 tx=0x7f855800b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.416+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f85640ce800 con 0x7f8568102ae0 2024-09-21T02:19:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.608+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8568102ef0 con 0x7f8568102ae0 2024-09-21T02:19:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.610+0000 7f855f7fe640 1 -- 172.21.15.16:0/2836863647 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25380 (secure 0 0 0) 0x7f85640d2be0 con 0x7f8568102ae0 2024-09-21T02:19:23.611 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-21T02:19:23.611 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"8372ca74-77bc-11ef-baf5-efdc52797490","created":"2024-09-21T01:56:00.153453+0000","modified":"2024-09-21T02:19:15.996330+0000","last_up_change":"2024-09-21T02:03:43.218892+0000","last_in_change":"2024-09-21T02:03:19.709915+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:59:52.388767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-21T02:04:29.848315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-21T02:04:32.294267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-21T02:04:38.676726+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2f8f1230-9fd6-4eb8-8804-c04765d2734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6803","nonce":1246966060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6805","nonce":1246966060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6809","nonce":1246966060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1246966060},{"type":"v1","addr":"172.21.15.16:6807","nonce":1246966060}]},"public_addr":"172.21.15.16:6803/1246966060","cluster_addr":"172.21.15.16:6805/1246966060","heartbeat_back_addr":"172.21.15.16:6809/1246966060","heartbeat_front_addr":"172.21.15.16:6807/1246966060","state":["exists","up"]},{"osd":1,"uuid":"354cac3a-106d-4c37-a1f3-dcbd1bdf142d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6801","nonce":4233413393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6803","nonce":4233413393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6807","nonce":4233413393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":4233413393},{"type":"v1","addr":"172.21.15.123:6805","nonce":4233413393}]},"public_addr":"172.21.15.123:6801/4233413393","cluster_addr":"172.21.15.123:6803/4233413393","heartbeat_back_addr":"172.21.15.123:6807/4233413393","heartbeat_front_addr":"172.21.15.123:6805/4233413393","state":["exists","up"]},{"osd":2,"uuid":"21a8f0b2-d4d2-4fdd-9302-7475215892e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6801","nonce":3882488819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6803","nonce":3882488819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6807","nonce":3882488819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3882488819},{"type":"v1","addr":"172.21.15.161:6805","nonce":3882488819}]},"public_addr":"172.21.15.161:6801/3882488819","cluster_addr":"172.21.15.161:6803/3882488819","heartbeat_back_addr":"172.21.15.161:6807/3882488819","heartbeat_front_addr":"172.21.15.161:6805/3882488819","state":["exists","up"]},{"osd":3,"uuid":"eb7c5f89-293f-4a90-9a26-3ba287b3f072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6811","nonce":2441025500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6813","nonce":2441025500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6817","nonce":2441025500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":2441025500},{"type":"v1","addr":"172.21.15.16:6815","nonce":2441025500}]},"public_addr":"172.21.15.16:6811/2441025500","cluster_addr":"172.21.15.16:6813/2441025500","heartbeat_back_addr":"172.21.15.16:6817/2441025500","heartbeat_front_addr":"172.21.15.16:6815/2441025500","state":["exists","up"]},{"osd":4,"uuid":"56d0113f-f7b9-463f-b027-41d611c683dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6809","nonce":1713366757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6811","nonce":1713366757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6815","nonce":1713366757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1713366757},{"type":"v1","addr":"172.21.15.123:6813","nonce":1713366757}]},"public_addr":"172.21.15.123:6809/1713366757","cluster_addr":"172.21.15.123:6811/1713366757","heartbeat_back_addr":"172.21.15.123:6815/1713366757","heartbeat_front_addr":"172.21.15.123:6813/1713366757","state":["exists","up"]},{"osd":5,"uuid":"6d1bb4b0-fc3a-488e-8b68-41360a19cef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6809","nonce":1422110321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6811","nonce":1422110321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6815","nonce":1422110321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1422110321},{"type":"v1","addr":"172.21.15.161:6813","nonce":1422110321}]},"public_addr":"172.21.15.161:6809/1422110321","cluster_addr":"172.21.15.161:6811/1422110321","heartbeat_back_addr":"172.21.15.161:6815/1422110321","heartbeat_front_addr":"172.21.15.161:6813/1422110321","state":["exists","up"]},{"osd":6,"uuid":"0c00ece7-8034-440b-ba71-50cc66f698a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6819","nonce":87656955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6821","nonce":87656955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6825","nonce":87656955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":87656955},{"type":"v1","addr":"172.21.15.16:6823","nonce":87656955}]},"public_addr":"172.21.15.16:6819/87656955","cluster_addr":"172.21.15.16:6821/87656955","heartbeat_back_addr":"172.21.15.16:6825/87656955","heartbeat_front_addr":"172.21.15.16:6823/87656955","state":["exists","up"]},{"osd":7,"uuid":"15352f53-b24f-4c15-8405-1971628489a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6817","nonce":271400775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6819","nonce":271400775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6823","nonce":271400775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":271400775},{"type":"v1","addr":"172.21.15.123:6821","nonce":271400775}]},"public_addr":"172.21.15.123:6817/271400775","cluster_addr":"172.21.15.123:6819/271400775","heartbeat_back_addr":"172.21.15.123:6823/271400775","heartbeat_front_addr":"172.21.15.123:6821/271400775","state":["exists","up"]},{"osd":8,"uuid":"71da0d14-803c-47fa-9a77-e44887e81da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6817","nonce":3772925422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6819","nonce":3772925422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6823","nonce":3772925422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3772925422},{"type":"v1","addr":"172.21.15.161:6821","nonce":3772925422}]},"public_addr":"172.21.15.161:6817/3772925422","cluster_addr":"172.21.15.161:6819/3772925422","heartbeat_back_addr":"172.21.15.161:6823/3772925422","heartbeat_front_addr":"172.21.15.161:6821/3772925422","state":["exists","up"]},{"osd":9,"uuid":"55a8fc26-a3a8-4a74-97d2-68218278bb36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6827","nonce":638537789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6829","nonce":638537789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6833","nonce":638537789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":638537789},{"type":"v1","addr":"172.21.15.16:6831","nonce":638537789}]},"public_addr":"172.21.15.16:6827/638537789","cluster_addr":"172.21.15.16:6829/638537789","heartbeat_back_addr":"172.21.15.16:6833/638537789","heartbeat_front_addr":"172.21.15.16:6831/638537789","state":["exists","up"]},{"osd":10,"uuid":"b9020569-c79b-468f-a9a0-78a812618945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6825","nonce":1402496387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6827","nonce":1402496387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6831","nonce":1402496387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1402496387},{"type":"v1","addr":"172.21.15.123:6829","nonce":1402496387}]},"public_addr":"172.21.15.123:6825/1402496387","cluster_addr":"172.21.15.123:6827/1402496387","heartbeat_back_addr":"172.21.15.123:6831/1402496387","heartbeat_front_addr":"172.21.15.123:6829/1402496387","state":["exists","up"]},{"osd":11,"uuid":"fc0b0eb6-640a-41fe-b884-b6752e5f75ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6825","nonce":1648324302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6827","nonce":1648324302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6831","nonce":1648324302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1648324302},{"type":"v1","addr":"172.21.15.161:6829","nonce":1648324302}]},"public_addr":"172.21.15.161:6825/1648324302","cluster_addr":"172.21.15.161:6827/1648324302","heartbeat_back_addr":"172.21.15.161:6831/1648324302","heartbeat_front_addr":"172.21.15.161:6829/1648324302","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:58:58.184664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:24.015421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:59:49.230533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:14.827581+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:00:40.730946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:06.546820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:32.957540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:01:58.839216+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:24.227782+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:02:50.270413+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:16.345443+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T02:03:41.170750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3095458517":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/2441291809":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6801/1946001099":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3677616942":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/4075907232":"2024-09-22T01:56:22.535497+0000","172.21.15.16:6800/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/2368126584":"2024-09-22T01:57:51.923009+0000","172.21.15.16:6801/1610958069":"2024-09-22T01:56:22.535497+0000","172.21.15.16:0/3423016963":"2024-09-22T01:56:49.870093+0000","172.21.15.16:0/3586667721":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3429917716":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/4160624590":"2024-09-22T01:57:51.923009+0000","172.21.15.16:0/3010596393":"2024-09-22T01:56:49.870093+0000","172.21.15.16:6800/1946001099":"2024-09-22T01:56:49.870093+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T02:19:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 msgr2=0x7f85540b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 0x7f85540b8bc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8568079fb0 tx=0x7f855800b390 comp rx=0 tx=0).stop 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 msgr2=0x7f856819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8564009230 tx=0x7f8564004270 comp rx=0 tx=0).stop 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 shutdown_connections 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f85540b66b0 0x7f85540b8bc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8568108d20 0x7f856819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85681083d0 0x7f856819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 --2- 172.21.15.16:0/2836863647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8568102ae0 0x7f856819a400 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 >> 172.21.15.16:0/2836863647 conn(0x7f8568079760 msgr2=0x7f8568106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 shutdown_connections 2024-09-21T02:19:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:23.612+0000 7f8570557640 1 -- 172.21.15.16:0/2836863647 wait complete. 2024-09-21T02:19:24.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:24 smithi016 ceph-mon[31245]: pgmap v733: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-21T02:19:24.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:24 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2836863647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:19:24.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:24 smithi123 ceph-mon[33453]: pgmap v733: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-21T02:19:24.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:24 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2836863647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:19:24.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs fail cephfs 2024-09-21T02:19:24.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:24 smithi161 ceph-mon[34061]: pgmap v733: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 818 B/s wr, 11 op/s 2024-09-21T02:19:24.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:24 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2836863647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T02:19:24.557 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 -- 172.21.15.16:0/419545701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1624104860 msgr2=0x7f1624104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 --2- 172.21.15.16:0/419545701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1624104860 0x7f1624104c70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1614009900 tx=0x7f161402a410 comp rx=0 tx=0).stop 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 -- 172.21.15.16:0/419545701 shutdown_connections 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 --2- 172.21.15.16:0/419545701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f162410de10 0x7f1624110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 --2- 172.21.15.16:0/419545701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624105290 0x7f162410d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 --2- 172.21.15.16:0/419545701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1624104860 0x7f1624104c70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 -- 172.21.15.16:0/419545701 >> 172.21.15.16:0/419545701 conn(0x7f1624079740 msgr2=0x7f1624079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 -- 172.21.15.16:0/419545701 shutdown_connections 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.043+0000 7f162a468640 1 -- 172.21.15.16:0/419545701 wait complete. 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.044+0000 7f162a468640 1 Processor -- start 2024-09-21T02:19:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.044+0000 7f162a468640 1 -- start start 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624104860 0x7f162419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1624105290 0x7f162419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1624113960 con 0x7f162410de10 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16241137e0 con 0x7f1624104860 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f162a468640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1624113ae0 con 0x7f1624105290 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51978/0 (socket says 172.21.15.16:51978) 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 -- 172.21.15.16:0/3868948338 learned_addr learned my addr 172.21.15.16:0/3868948338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f161bfff640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1624105290 0x7f162419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1623fff640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624104860 0x7f162419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 -- 172.21.15.16:0/3868948338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1624105290 msgr2=0x7f162419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1624105290 0x7f162419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 -- 172.21.15.16:0/3868948338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624104860 msgr2=0x7f162419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624104860 0x7f162419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 -- 172.21.15.16:0/3868948338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16241a03a0 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.045+0000 7f1628a05640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f161c00eb10 tx=0x7f161c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.046+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c004b20 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.046+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f161c004cc0 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.046+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16241a0690 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.046+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c010690 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.046+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16241a6df0 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.047+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f162406c1e0 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.048+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f161c005000 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.049+0000 7f1621ffb640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 0x7f16100b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.049+0000 7f1623fff640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 0x7f16100b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.049+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9482+0+0 (secure 0 0 0) 0x7f161c0e82b0 con 0x7f162410de10 2024-09-21T02:19:25.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.050+0000 7f1623fff640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 0x7f16100b8b90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1614006010 tx=0x7f1614032c80 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.052+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f161c056710 con 0x7f162410de10 2024-09-21T02:19:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.279+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1624104c70 con 0x7f162410de10 2024-09-21T02:19:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.282+0000 7f1621ffb640 1 -- 172.21.15.16:0/3868948338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f161c0ae3e0 con 0x7f162410de10 2024-09-21T02:19:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-21T02:19:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 msgr2=0x7f16100b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 0x7f16100b8b90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1614006010 tx=0x7f1614032c80 comp rx=0 tx=0).stop 2024-09-21T02:19:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 msgr2=0x7f162419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f161c00eb10 tx=0x7f161c00c2d0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 shutdown_connections 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f16100b6680 0x7f16100b8b90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f162410de10 0x7f162419fc70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1624105290 0x7f162419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 --2- 172.21.15.16:0/3868948338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1624104860 0x7f162419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.286+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 >> 172.21.15.16:0/3868948338 conn(0x7f1624079740 msgr2=0x7f16240ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.287+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 shutdown_connections 2024-09-21T02:19:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:25.287+0000 7f162a468640 1 -- 172.21.15.16:0/3868948338 wait complete. 2024-09-21T02:19:25.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a[31241]: 2024-09-21T02:19:25.280+0000 7f1b01f62640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-mds[74123]: -- 172.21.15.16:0/2558181759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636b7b45c20 con 0x5636bbced180 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: starting mds.g at 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-mds[74503]: -- 172.21.15.16:0/2634624865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d416c35860 con 0x55d41addd180 2024-09-21T02:19:25.659 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:19:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: starting mds.a at 2024-09-21T02:19:25.665 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.665 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.665 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: starting mds.l at 2024-09-21T02:19:25.665 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.665 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-mds[63927]: -- 172.21.15.161:0/3608907384 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557219b3b860 con 0x55721dce3a80 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: starting mds.c at 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-mds[63655]: -- 172.21.15.161:0/2955061461 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ef8327860 con 0x555efc4cfa80 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: starting mds.f at 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.666 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-mds[64193]: -- 172.21.15.161:0/2238280800 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f545c13860 con 0x55f549dbb600 2024-09-21T02:19:25.667 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:19:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: starting mds.i at 2024-09-21T02:19:25.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[61530]: -- 172.21.15.123:0/2718445598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623e9a5b860 con 0x5623edc03600 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: starting mds.k at 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[61807]: -- 172.21.15.123:0/2489314671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2cef79860 con 0x55b2d3121a80 2024-09-21T02:19:25.771 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: starting mds.e at 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[62350]: -- 172.21.15.123:0/3325281575 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e893cf860 con 0x562e8d577180 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: starting mds.b at 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: ignoring --setuser ceph since I am not root 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: ignoring --setgroup ceph since I am not root 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[62082]: --2- 172.21.15.123:0/701862813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5579f2c57600 0x5579f3846580 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x5579f2c68b70 tx=0x5579f2b8e540 comp rx=0 tx=0).write_message sending message m=0x5579f2b1c540 seq=3 auth(proto 2 2 bytes epoch 0) 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[62082]: -- 172.21.15.123:0/701862813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5579eeab8b60 con 0x5579f2c57600 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-mds[62082]: -- 172.21.15.123:0/701862813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579eeaafc20 con 0x5579f2c57600 2024-09-21T02:19:25.772 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:19:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: starting mds.h at 2024-09-21T02:19:26.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-21T02:19:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: pgmap v734: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T02:19:26.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T02:19:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T02:19:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T02:19:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T02:19:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T02:19:26.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:26 smithi123 ceph-mon[33453]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T02:19:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: pgmap v734: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T02:19:26.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:26 smithi016 ceph-mon[31245]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: pgmap v734: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd={"prefix": "fs fail", "fs_name": "cephfs"} : dispatch 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3868948338' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: osdmap e101: 12 total, 12 up, 12 in 2024-09-21T02:19:26.415 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:26 smithi161 ceph-mon[34061]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 -- 172.21.15.16:0/4191123905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 msgr2=0x7ff3bc102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/4191123905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc102650 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009900 tx=0x7ff3b802a410 comp rx=0 tx=0).stop 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 -- 172.21.15.16:0/4191123905 shutdown_connections 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/4191123905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 0x7ff3bc10ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/4191123905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 0x7ff3bc107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/4191123905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc102650 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 -- 172.21.15.16:0/4191123905 >> 172.21.15.16:0/4191123905 conn(0x7ff3bc079f90 msgr2=0x7ff3bc07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 -- 172.21.15.16:0/4191123905 shutdown_connections 2024-09-21T02:19:26.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.779+0000 7ff3c33d5640 1 -- 172.21.15.16:0/4191123905 wait complete. 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 Processor -- start 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 -- start start 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 0x7ff3bc19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 0x7ff3bc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc1117c0 con 0x7ff3bc102240 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc111640 con 0x7ff3bc107b80 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c33d5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc111940 con 0x7ff3bc1084d0 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c1171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c1171640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52032/0 (socket says 172.21.15.16:52032) 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c1171640 1 -- 172.21.15.16:0/612402833 learned_addr learned my addr 172.21.15.16:0/612402833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.780+0000 7ff3c0970640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 0x7ff3bc19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1972640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 0x7ff3bc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 -- 172.21.15.16:0/612402833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 msgr2=0x7ff3bc19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 0x7ff3bc19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 -- 172.21.15.16:0/612402833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 msgr2=0x7ff3bc19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 0x7ff3bc19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 -- 172.21.15.16:0/612402833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc1a0390 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c1171640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009230 tx=0x7ff3b8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b803f030 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc1a0620 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3b8003a80 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8035070 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.781+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc1a6de0 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.782+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3bc102650 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.784+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7ff3b8042030 con 0x7ff3bc102240 2024-09-21T02:19:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.784+0000 7ff3b27fc640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 0x7ff3a80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.785+0000 7ff3c0970640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 0x7ff3a80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.785+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9824+0+0 (secure 0 0 0) 0x7ff3b81083b0 con 0x7ff3bc102240 2024-09-21T02:19:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.785+0000 7ff3c0970640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 0x7ff3a80b8b90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc104260 tx=0x7ff3ac00b390 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:26.789+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff3b80ce3b0 con 0x7ff3bc102240 2024-09-21T02:19:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.014+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff3bc10a2d0 con 0x7ff3bc102240 2024-09-21T02:19:27.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.289+0000 7ff3b27fc640 1 -- 172.21.15.16:0/612402833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) ==== 104+0+0 (secure 0 0 0) 0x7ff3b80d2790 con 0x7ff3bc102240 2024-09-21T02:19:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 msgr2=0x7ff3a80b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 0x7ff3a80b8b90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc104260 tx=0x7ff3ac00b390 comp rx=0 tx=0).stop 2024-09-21T02:19:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 msgr2=0x7ff3bc19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009230 tx=0x7ff3b8004270 comp rx=0 tx=0).stop 2024-09-21T02:19:27.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 shutdown_connections 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7ff3a80b6680 0x7ff3a80b8b90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3bc1084d0 0x7ff3bc19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3bc107b80 0x7ff3bc19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 --2- 172.21.15.16:0/612402833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3bc102240 0x7ff3bc19a420 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 >> 172.21.15.16:0/612402833 conn(0x7ff3bc079f90 msgr2=0x7ff3bc105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 shutdown_connections 2024-09-21T02:19:27.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:27.292+0000 7ff3c33d5640 1 -- 172.21.15.16:0/612402833 wait complete. 2024-09-21T02:19:27.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6832/3632894453,v1:172.21.15.123:6836/3632894453] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6833/1438149980,v1:172.21.15.123:6837/1438149980] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6835/3001160226,v1:172.21.15.123:6839/3001160226] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.123:6834/1192763969,v1:172.21.15.123:6838/1192763969] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6832/746671385,v1:172.21.15.161:6833/746671385] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6834/1065750799,v1:172.21.15.161:6835/1065750799] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6836/3678375125,v1:172.21.15.161:6837/3678375125] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.161:6838/3841799705,v1:172.21.15.161:6839/3841799705] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6839/269673577,v1:172.21.15.16:6841/269673577] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: mds.? [v2:172.21.15.16:6838/2620425170,v1:172.21.15.16:6840/2620425170] up:boot 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:19:27.659 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: pgmap v736: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-21T02:19:27.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:27 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6832/3632894453,v1:172.21.15.123:6836/3632894453] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6833/1438149980,v1:172.21.15.123:6837/1438149980] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6835/3001160226,v1:172.21.15.123:6839/3001160226] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.123:6834/1192763969,v1:172.21.15.123:6838/1192763969] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6832/746671385,v1:172.21.15.161:6833/746671385] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6834/1065750799,v1:172.21.15.161:6835/1065750799] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6836/3678375125,v1:172.21.15.161:6837/3678375125] up:boot 2024-09-21T02:19:27.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.161:6838/3841799705,v1:172.21.15.161:6839/3841799705] up:boot 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6839/269673577,v1:172.21.15.16:6841/269673577] up:boot 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: mds.? [v2:172.21.15.16:6838/2620425170,v1:172.21.15.16:6840/2620425170] up:boot 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:19:27.666 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: pgmap v736: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-21T02:19:27.667 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:27 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6832/3632894453,v1:172.21.15.123:6836/3632894453] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6833/1438149980,v1:172.21.15.123:6837/1438149980] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6835/3001160226,v1:172.21.15.123:6839/3001160226] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.123:6834/1192763969,v1:172.21.15.123:6838/1192763969] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6832/746671385,v1:172.21.15.161:6833/746671385] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6834/1065750799,v1:172.21.15.161:6835/1065750799] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6836/3678375125,v1:172.21.15.161:6837/3678375125] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.161:6838/3841799705,v1:172.21.15.161:6839/3841799705] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6839/269673577,v1:172.21.15.16:6841/269673577] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: mds.? [v2:172.21.15.16:6838/2620425170,v1:172.21.15.16:6840/2620425170] up:boot 2024-09-21T02:19:27.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "a"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "g"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "e"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "k"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "b"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "c"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "h"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "l"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "i"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='mgr.14223 172.21.15.16:0/4249440018' entity='mgr.x' cmd={"prefix": "mds metadata", "who": "f"} : dispatch 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: pgmap v736: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-09-21T02:19:27.772 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:27 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd={"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} : dispatch 2024-09-21T02:19:28.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-21T02:19:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:28.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:28 smithi016 ceph-mon[31245]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T02:19:28.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:28 smithi016 ceph-mon[31245]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T02:19:28.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:28 smithi016 ceph-mon[31245]: Cluster is now healthy 2024-09-21T02:19:28.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:28 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:19:28.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:28 smithi016 ceph-mon[31245]: fsmap 12 up:standby 2024-09-21T02:19:28.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:28 smithi161 ceph-mon[34061]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T02:19:28.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:28 smithi161 ceph-mon[34061]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T02:19:28.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:28 smithi161 ceph-mon[34061]: Cluster is now healthy 2024-09-21T02:19:28.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:28 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:19:28.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:28 smithi161 ceph-mon[34061]: fsmap 12 up:standby 2024-09-21T02:19:28.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:28 smithi123 ceph-mon[33453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T02:19:28.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:28 smithi123 ceph-mon[33453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T02:19:28.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:28 smithi123 ceph-mon[33453]: Cluster is now healthy 2024-09-21T02:19:28.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:28 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/612402833' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-21T02:19:28.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:28 smithi123 ceph-mon[33453]: fsmap 12 up:standby 2024-09-21T02:19:28.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 -- 172.21.15.16:0/2580164419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 msgr2=0x7f67241098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:28.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 --2- 172.21.15.16:0/2580164419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f67241098d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6714009900 tx=0x7f671402a3f0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 -- 172.21.15.16:0/2580164419 shutdown_connections 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 --2- 172.21.15.16:0/2580164419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 0x7f6724110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 --2- 172.21.15.16:0/2580164419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f67241098d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 --2- 172.21.15.16:0/2580164419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 0x7f6724103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 -- 172.21.15.16:0/2580164419 >> 172.21.15.16:0/2580164419 conn(0x7f67240ff5d0 msgr2=0x7f6724101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 -- 172.21.15.16:0/2580164419 shutdown_connections 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.784+0000 7f672c7b2640 1 -- 172.21.15.16:0/2580164419 wait complete. 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 Processor -- start 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 -- start start 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 0x7f672419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 0x7f672419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:28.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6724113980 con 0x7f67241094c0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6724113800 con 0x7f6724109e10 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f672c7b2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6724113b00 con 0x7f6724103b80 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f6729d4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f6729d4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53716/0 (socket says 172.21.15.16:53716) 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.785+0000 7f6729d4d640 1 -- 172.21.15.16:0/1194390042 learned_addr learned my addr 172.21.15.16:0/1194390042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f672a54e640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 0x7f672419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f672ad4f640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 0x7f672419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 -- 172.21.15.16:0/1194390042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 msgr2=0x7f672419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 0x7f672419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 -- 172.21.15.16:0/1194390042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 msgr2=0x7f672419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 0x7f672419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 -- 172.21.15.16:0/1194390042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67241a02e0 con 0x7f67241094c0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f6729d4d640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6714005940 tx=0x7f67140040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671403f030 con 0x7f67241094c0 2024-09-21T02:19:28.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67241a0570 con 0x7f67241094c0 2024-09-21T02:19:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6714003a80 con 0x7f67241094c0 2024-09-21T02:19:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6714035070 con 0x7f67241094c0 2024-09-21T02:19:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.786+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67241a0aa0 con 0x7f67241094c0 2024-09-21T02:19:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.788+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6724103f90 con 0x7f67241094c0 2024-09-21T02:19:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.789+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f6714042030 con 0x7f67241094c0 2024-09-21T02:19:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.790+0000 7f671b7fe640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 0x7f67100b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.790+0000 7f672a54e640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 0x7f67100b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.790+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9824+0+0 (secure 0 0 0) 0x7f6714108550 con 0x7f67241094c0 2024-09-21T02:19:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.790+0000 7f672a54e640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 0x7f67100b8bc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6720005d00 tx=0x7f672000b410 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:28.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.793+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f67241a0aa0 con 0x7f67241094c0 2024-09-21T02:19:28.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:28.988+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f672419b560 con 0x7f67241094c0 2024-09-21T02:19:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.297+0000 7f671b7fe640 1 -- 172.21.15.16:0/1194390042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) ==== 181+0+0 (secure 0 0 0) 0x7f67140ce590 con 0x7f67241094c0 2024-09-21T02:19:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-09-21T02:19:29.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.301+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 msgr2=0x7f67100b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:29.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.301+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 0x7f67100b8bc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6720005d00 tx=0x7f672000b410 comp rx=0 tx=0).stop 2024-09-21T02:19:29.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.301+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 msgr2=0x7f672419a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:29.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.301+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6714005940 tx=0x7f67140040e0 comp rx=0 tx=0).stop 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 shutdown_connections 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f67100b66b0 0x7f67100b8bc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6724109e10 0x7f672419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67241094c0 0x7f672419a9d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 --2- 172.21.15.16:0/1194390042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6724103b80 0x7f672419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 >> 172.21.15.16:0/1194390042 conn(0x7f67240ff5d0 msgr2=0x7f67241019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 shutdown_connections 2024-09-21T02:19:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:29.302+0000 7f672c7b2640 1 -- 172.21.15.16:0/1194390042 wait complete. 2024-09-21T02:19:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:29 smithi016 ceph-mon[31245]: pgmap v737: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:29.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:29 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:29.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:29 smithi161 ceph-mon[34061]: pgmap v737: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:29.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:29 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:29.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:29 smithi123 ceph-mon[33453]: pgmap v737: 129 pgs: 129 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-21T02:19:29.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:29 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:30.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-21T02:19:30.405 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:30 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:30.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:30 smithi016 ceph-mon[31245]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T02:19:30.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:30 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:30.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:30 smithi161 ceph-mon[34061]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T02:19:30.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:30 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/1194390042' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:30.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:30 smithi123 ceph-mon[33453]: osdmap e102: 12 total, 12 up, 12 in 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 -- 172.21.15.16:0/656134660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9330102a90 msgr2=0x7f9330102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 --2- 172.21.15.16:0/656134660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9330102a90 0x7f9330102ea0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9320009900 tx=0x7f932002a410 comp rx=0 tx=0).stop 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 -- 172.21.15.16:0/656134660 shutdown_connections 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 --2- 172.21.15.16:0/656134660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330108d20 0x7f933010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 --2- 172.21.15.16:0/656134660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93301083d0 0x7f93301087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 --2- 172.21.15.16:0/656134660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9330102a90 0x7f9330102ea0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 -- 172.21.15.16:0/656134660 >> 172.21.15.16:0/656134660 conn(0x7f9330079750 msgr2=0x7f9330079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.891+0000 7f9335d2c640 1 -- 172.21.15.16:0/656134660 shutdown_connections 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.892+0000 7f9335d2c640 1 -- 172.21.15.16:0/656134660 wait complete. 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.892+0000 7f9335d2c640 1 Processor -- start 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.892+0000 7f9335d2c640 1 -- start start 2024-09-21T02:19:30.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330102a90 0x7f9330198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9330108d20 0x7f9330199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9330112840 con 0x7f93301083d0 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93301126c0 con 0x7f9330108d20 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f9335d2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f93301129c0 con 0x7f9330102a90 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53748/0 (socket says 172.21.15.16:53748) 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 -- 172.21.15.16:0/2898586224 learned_addr learned my addr 172.21.15.16:0/2898586224 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932f7fe640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330102a90 0x7f9330198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932ffff640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9330108d20 0x7f9330199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 -- 172.21.15.16:0/2898586224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330102a90 msgr2=0x7f9330198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330102a90 0x7f9330198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 -- 172.21.15.16:0/2898586224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9330108d20 msgr2=0x7f9330199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:30.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9330108d20 0x7f9330199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 -- 172.21.15.16:0/2898586224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93301a8690 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.893+0000 7f932effd640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f931800d770 tx=0x7f931800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.894+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318017070 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.894+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93301a8980 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.894+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9318009c30 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.894+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931800f620 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.894+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93301a8ec0 con 0x7f93301083d0 2024-09-21T02:19:30.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.895+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9330103d10 con 0x7f93301083d0 2024-09-21T02:19:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.896+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f931801a070 con 0x7f93301083d0 2024-09-21T02:19:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.897+0000 7f932cff9640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 0x7f931c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.897+0000 7f932f7fe640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 0x7f931c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.897+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9270+0+0 (secure 0 0 0) 0x7f93180e73c0 con 0x7f93301083d0 2024-09-21T02:19:30.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.898+0000 7f932f7fe640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 0x7f931c0b8b90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9330079c00 tx=0x7f93200058d0 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:30.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:30.900+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9318009de0 con 0x7f93301083d0 2024-09-21T02:19:31.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.093+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9330102ea0 con 0x7f93301083d0 2024-09-21T02:19:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.300+0000 7f932cff9640 1 -- 172.21.15.16:0/2898586224 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) ==== 169+0+0 (secure 0 0 0) 0x7f93180b4e20 con 0x7f93301083d0 2024-09-21T02:19:31.301 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-09-21T02:19:31.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.303+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 msgr2=0x7f931c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.303+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 0x7f931c0b8b90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9330079c00 tx=0x7f93200058d0 comp rx=0 tx=0).stop 2024-09-21T02:19:31.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.303+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 msgr2=0x7f9330199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.303+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f931800d770 tx=0x7f931800dc30 comp rx=0 tx=0).stop 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 shutdown_connections 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f931c0b6680 0x7f931c0b8b90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9330108d20 0x7f9330199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93301083d0 0x7f9330199520 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 --2- 172.21.15.16:0/2898586224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9330102a90 0x7f9330198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:31.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.304+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 >> 172.21.15.16:0/2898586224 conn(0x7f9330079750 msgr2=0x7f93301015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.305+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 shutdown_connections 2024-09-21T02:19:31.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:31.305+0000 7f9335d2c640 1 -- 172.21.15.16:0/2898586224 wait complete. 2024-09-21T02:19:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:31 smithi016 ceph-mon[31245]: pgmap v739: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:31.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:31 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:31.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:31 smithi161 ceph-mon[34061]: pgmap v739: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:31.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:31 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:31 smithi123 ceph-mon[33453]: pgmap v739: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:31.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:31 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:32.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-21T02:19:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/mon.a/config 2024-09-21T02:19:32.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:32 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:32.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:32 smithi016 ceph-mon[31245]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T02:19:32.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:32 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:32.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:32 smithi161 ceph-mon[34061]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T02:19:32.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:32 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/2898586224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:32.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:32 smithi123 ceph-mon[33453]: osdmap e103: 12 total, 12 up, 12 in 2024-09-21T02:19:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 -- 172.21.15.16:0/4264462441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 msgr2=0x7f8c4c110680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/4264462441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c110680 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8c44009900 tx=0x7f8c4402a410 comp rx=0 tx=0).stop 2024-09-21T02:19:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 -- 172.21.15.16:0/4264462441 shutdown_connections 2024-09-21T02:19:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/4264462441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c110680 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/4264462441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 0x7f8c4c109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/4264462441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 -- 172.21.15.16:0/4264462441 >> 172.21.15.16:0/4264462441 conn(0x7f8c4c079f90 msgr2=0x7f8c4c07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.792+0000 7f8c53d2b640 1 -- 172.21.15.16:0/4264462441 shutdown_connections 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 -- 172.21.15.16:0/4264462441 wait complete. 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 Processor -- start 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 -- start start 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 0x7f8c4c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1139c0 con 0x7f8c4c109e50 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113840 con 0x7f8c4c109500 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.793+0000 7f8c53d2b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113b40 con 0x7f8c4c101fc0 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53764/0 (socket says 172.21.15.16:53764) 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 -- 172.21.15.16:0/3501982586 learned_addr learned my addr 172.21.15.16:0/3501982586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c51ac7640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c512c6640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 0x7f8c4c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 -- 172.21.15.16:0/3501982586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 msgr2=0x7f8c4c19c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c19c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 -- 172.21.15.16:0/3501982586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 msgr2=0x7f8c4c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 0x7f8c4c19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 -- 172.21.15.16:0/3501982586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c1a2580 con 0x7f8c4c109e50 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c51ac7640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c522c8640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005ab0 tx=0x7f8c440040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4403f030 con 0x7f8c4c109e50 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c1a2810 con 0x7f8c4c109e50 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8c44003a80 con 0x7f8c4c109e50 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44035070 con 0x7f8c4c109e50 2024-09-21T02:19:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.794+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c1a8fd0 con 0x7f8c4c109e50 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.796+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c06b930 con 0x7f8c4c109e50 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.797+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 154643+0+0 (secure 0 0 0) 0x7f8c44042030 con 0x7f8c4c109e50 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.798+0000 7f8c4affd640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 0x7f8c340b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.798+0000 7f8c51ac7640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 0x7f8c340b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.798+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8711+0+0 (secure 0 0 0) 0x7f8c44108200 con 0x7f8c4c109e50 2024-09-21T02:19:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.799+0000 7f8c51ac7640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 0x7f8c340b8b90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005fd0 tx=0x7f8c3c009520 comp rx=0 tx=0).ready entity=mgr.14223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T02:19:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.801+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8c440ce630 con 0x7f8c4c109e50 2024-09-21T02:19:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:32.991+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8c4c1a91a0 con 0x7f8c4c109e50 2024-09-21T02:19:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.316+0000 7f8c4affd640 1 -- 172.21.15.16:0/3501982586 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v104) ==== 178+0+0 (secure 0 0 0) 0x7f8c440d2a10 con 0x7f8c4c109e50 2024-09-21T02:19:33.317 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 msgr2=0x7f8c340b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 0x7f8c340b8b90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005fd0 tx=0x7f8c3c009520 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 msgr2=0x7f8c4c1a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005ab0 tx=0x7f8c440040e0 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 shutdown_connections 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:6800/3647438626,v1:172.21.15.16:6801/3647438626] conn(0x7f8c340b6680 0x7f8c340b8b90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c4c109e50 0x7f8c4c1a1e50 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c4c109500 0x7f8c4c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 --2- 172.21.15.16:0/3501982586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c4c101fc0 0x7f8c4c19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 >> 172.21.15.16:0/3501982586 conn(0x7f8c4c079f90 msgr2=0x7f8c4c105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T02:19:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 shutdown_connections 2024-09-21T02:19:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-21T02:19:33.320+0000 7f8c53d2b640 1 -- 172.21.15.16:0/3501982586 wait complete. 2024-09-21T02:19:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:33 smithi016 ceph-mon[31245]: pgmap v741: 33 pgs: 33 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:33.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:33 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:33.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:33 smithi161 ceph-mon[34061]: pgmap v741: 33 pgs: 33 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:33.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:33 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:33.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:33 smithi123 ceph-mon[33453]: pgmap v741: 33 pgs: 33 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-21T02:19:33.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:33 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T02:19:34.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:19:34.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:19:34.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:19:34.507 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T02:19:34.507 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T02:19:34.539 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T02:19:34.557 DEBUG:teuthology.orchestra.run.smithi161:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T02:19:34.578 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T02:19:34.578 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T02:19:34.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a 2024-09-21T02:19:34.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:34 smithi016 ceph-mon[31245]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:34.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:34 smithi016 ceph-mon[31245]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T02:19:34.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:34 smithi161 ceph-mon[34061]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:34.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:34 smithi161 ceph-mon[34061]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T02:19:34.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:34 smithi123 ceph-mon[33453]: from='client.? 172.21.15.16:0/3501982586' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T02:19:34.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:34 smithi123 ceph-mon[33453]: osdmap e104: 12 total, 12 up, 12 in 2024-09-21T02:19:34.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:34 smithi016 systemd[1]: Stopping Ceph mon.a for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a[31241]: 2024-09-21T02:19:35.079+0000 7f1b0776d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:19:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a[31241]: 2024-09-21T02:19:35.079+0000 7f1b0776d640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T02:19:35.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:35 smithi016 podman[104540]: 2024-09-21 02:19:35.21985211 +0000 UTC m=+0.286623947 container died e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:19:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:35 smithi016 podman[104540]: 2024-09-21 02:19:35.892727516 +0000 UTC m=+0.959499347 container cleanup e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T02:19:36.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:35 smithi016 bash[104540]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a 2024-09-21T02:19:36.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Sep 21 02:19:36 smithi016 podman[104551]: 2024-09-21 02:19:36.361938682 +0000 UTC m=+1.132073038 container remove e5de3bb59457c92de3cec08f8b7d8bd63e98887763e8c5a46128183755437152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:19:36.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.a.service' 2024-09-21T02:19:36.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:36.936 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T02:19:36.936 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-21T02:19:36.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.b 2024-09-21T02:19:37.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:37 smithi123 systemd[1]: Stopping Ceph mon.b for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:39.520 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b[33449]: 2024-09-21T02:19:39.237+0000 7fea66351640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:19:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b[33449]: 2024-09-21T02:19:39.237+0000 7fea66351640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T02:19:39.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:39 smithi123 podman[65826]: 2024-09-21 02:19:39.380988713 +0000 UTC m=+2.088750464 container died 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T02:19:44.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:43 smithi123 podman[65826]: 2024-09-21 02:19:43.964687633 +0000 UTC m=+6.672449375 container cleanup 407c11fb6f06504a57e8d91f3debef6581226fbbc76e20c9f0286e83196bd0b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:19:44.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Sep 21 02:19:43 smithi123 bash[65826]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-b 2024-09-21T02:19:44.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.b.service' 2024-09-21T02:19:44.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:44.483 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-21T02:19:44.483 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-21T02:19:44.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.c 2024-09-21T02:19:44.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:44 smithi161 systemd[1]: Stopping Ceph mon.c for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:45.387 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c[34057]: 2024-09-21T02:19:44.918+0000 7f39c3879640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:19:45.387 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c[34057]: 2024-09-21T02:19:44.918+0000 7f39c3879640 -1 mon.c@1(probing) e3 *** Got Signal Terminated *** 2024-09-21T02:19:45.387 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 podman[67762]: 2024-09-21 02:19:45.041181213 +0000 UTC m=+0.309789805 container died c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD) 2024-09-21T02:19:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 podman[67762]: 2024-09-21 02:19:45.485761664 +0000 UTC m=+0.754370256 container cleanup c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T02:19:45.665 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 bash[67762]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c 2024-09-21T02:19:45.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.c.service' 2024-09-21T02:19:45.997 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 podman[67776]: 2024-09-21 02:19:45.729109462 +0000 UTC m=+0.684732812 container remove c6d18eb1eb6889882502afccae6535080c2b9e02636acd9ff9babf7cb8383404 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T02:19:45.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 systemd[1]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T02:19:45.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 systemd[1]: Stopped Ceph mon.c for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:19:45.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Sep 21 02:19:45 smithi161 systemd[1]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mon.c.service: Consumed 14.166s CPU time. 2024-09-21T02:19:46.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:46.025 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-21T02:19:46.025 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T02:19:46.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x 2024-09-21T02:19:46.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:46 smithi016 systemd[1]: Stopping Ceph mgr.x for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:48.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x[31470]: 2024-09-21T02:19:47.829+0000 7f59ed793640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:19:48.159 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:47 smithi016 podman[104648]: 2024-09-21 02:19:47.870966785 +0000 UTC m=+1.647132086 container died 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T02:19:53.155 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:52 smithi016 podman[104648]: 2024-09-21 02:19:52.740891188 +0000 UTC m=+6.517056479 container cleanup 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T02:19:53.156 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:52 smithi016 bash[104648]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x 2024-09-21T02:19:53.408 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Sep 21 02:19:53 smithi016 podman[104672]: 2024-09-21 02:19:53.155472414 +0000 UTC m=+5.281918823 container remove 8cd744d910c16dada3491e729a686e5c75ada2bfae9ec5bc6b40f3bee2fa76c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-x, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:19:53.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-21T02:19:53.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:53.470 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T02:19:53.470 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T02:19:53.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.y 2024-09-21T02:19:53.770 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 02:19:53 smithi123 systemd[1]: Stopping Ceph mgr.y for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:54.104 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 02:19:53 smithi123 podman[65946]: 2024-09-21 02:19:53.823314449 +0000 UTC m=+0.162211312 container died caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T02:19:54.502 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 02:19:54 smithi123 podman[65946]: 2024-09-21 02:19:54.202916259 +0000 UTC m=+0.541813131 container cleanup caaf2d42c60bc29b69960b8a25b4285f7e9fed1a4c78e0964773f8ce9ed0951d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-21T02:19:54.502 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Sep 21 02:19:54 smithi123 bash[65946]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-y 2024-09-21T02:19:54.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-21T02:19:54.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:54.706 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T02:19:54.706 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-21T02:19:54.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.z 2024-09-21T02:19:55.037 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 02:19:54 smithi161 systemd[1]: Stopping Ceph mgr.z for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:55.292 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 02:19:55 smithi161 podman[67894]: 2024-09-21 02:19:55.036081637 +0000 UTC m=+0.130084972 container died 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T02:19:55.665 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 02:19:55 smithi161 podman[67894]: 2024-09-21 02:19:55.390514928 +0000 UTC m=+0.484518270 container cleanup 6870ba08b8c3f90d8f96267de3a606acb88ea58cfc3f9e199fa8088c969b8b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T02:19:55.665 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Sep 21 02:19:55 smithi161 bash[67894]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mgr-z 2024-09-21T02:19:55.862 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mgr.z.service' 2024-09-21T02:19:55.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:19:55.898 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-21T02:19:55.898 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T02:19:55.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.0 2024-09-21T02:19:56.226 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:19:55 smithi016 systemd[1]: Stopping Ceph osd.0 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:19:56.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:19:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T02:19:56.225+0000 7f8f86102640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:19:56.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:19:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T02:19:56.225+0000 7f8f86102640 -1 osd.0 104 *** Got signal Terminated *** 2024-09-21T02:19:56.659 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:19:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0[48669]: 2024-09-21T02:19:56.225+0000 7f8f86102640 -1 osd.0 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:02.285 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:01 smithi016 podman[104779]: 2024-09-21 02:20:01.959088837 +0000 UTC m=+5.852108005 container died 6f5e5e73913dac5f633fb23bf544a743d23bd57cda87098ab3e9e1f6c723ce36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T02:20:02.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:02 smithi016 podman[104779]: 2024-09-21 02:20:02.476278356 +0000 UTC m=+6.369297564 container cleanup 6f5e5e73913dac5f633fb23bf544a743d23bd57cda87098ab3e9e1f6c723ce36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T02:20:02.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:02 smithi016 bash[104779]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0 2024-09-21T02:20:03.064 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:02 smithi016 podman[104796]: 2024-09-21 02:20:02.744724655 +0000 UTC m=+0.776975386 container remove 6f5e5e73913dac5f633fb23bf544a743d23bd57cda87098ab3e9e1f6c723ce36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T02:20:03.332 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104887]: 2024-09-21 02:20:02.967082694 +0000 UTC m=+0.016437920 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:03.333 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104887]: 2024-09-21 02:20:03.079043857 +0000 UTC m=+0.128399075 container create 1dc20bec541de7f0daca741fd5e1c8d3ed61984f2018c892a341da411221ca8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T02:20:03.333 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104887]: 2024-09-21 02:20:03.332336874 +0000 UTC m=+0.381692107 container init 1dc20bec541de7f0daca741fd5e1c8d3ed61984f2018c892a341da411221ca8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T02:20:03.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104887]: 2024-09-21 02:20:03.336624269 +0000 UTC m=+0.385979490 container start 1dc20bec541de7f0daca741fd5e1c8d3ed61984f2018c892a341da411221ca8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T02:20:03.658 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104887]: 2024-09-21 02:20:03.449114102 +0000 UTC m=+0.498469331 container attach 1dc20bec541de7f0daca741fd5e1c8d3ed61984f2018c892a341da411221ca8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T02:20:04.036 INFO:journalctl@ceph.osd.0.smithi016.stdout:Sep 21 02:20:03 smithi016 podman[104910]: 2024-09-21 02:20:03.751348777 +0000 UTC m=+0.017789313 container died 1dc20bec541de7f0daca741fd5e1c8d3ed61984f2018c892a341da411221ca8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T02:20:04.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.0.service' 2024-09-21T02:20:04.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:04.416 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T02:20:04.417 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T02:20:04.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.1 2024-09-21T02:20:04.736 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:04 smithi123 systemd[1]: Stopping Ceph osd.1 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:05.020 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T02:20:04.735+0000 7f86fca30640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:05.021 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T02:20:04.735+0000 7f86fca30640 -1 osd.1 104 *** Got signal Terminated *** 2024-09-21T02:20:05.021 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1[38166]: 2024-09-21T02:20:04.735+0000 7f86fca30640 -1 osd.1 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:10.858 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:10 smithi123 podman[66070]: 2024-09-21 02:20:10.5671049 +0000 UTC m=+5.947569850 container died bc1d5a158bc4c1c5b2472ed18162e32a5781674fda46f59c8b500e4c1e92d4ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:20:11.209 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:10 smithi123 podman[66070]: 2024-09-21 02:20:10.940208883 +0000 UTC m=+6.320673830 container cleanup bc1d5a158bc4c1c5b2472ed18162e32a5781674fda46f59c8b500e4c1e92d4ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T02:20:11.209 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:10 smithi123 bash[66070]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1 2024-09-21T02:20:11.520 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66088]: 2024-09-21 02:20:11.208573895 +0000 UTC m=+0.644951697 container remove bc1d5a158bc4c1c5b2472ed18162e32a5781674fda46f59c8b500e4c1e92d4ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:11.879 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66175]: 2024-09-21 02:20:11.463705937 +0000 UTC m=+0.015826627 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:11.879 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66175]: 2024-09-21 02:20:11.593003949 +0000 UTC m=+0.145124667 container create 14794681295d882d6dcc2cc005243dee8a5d0f54b3eba47eb93ab0f6631a4088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T02:20:12.271 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66175]: 2024-09-21 02:20:11.879217352 +0000 UTC m=+0.431338042 container init 14794681295d882d6dcc2cc005243dee8a5d0f54b3eba47eb93ab0f6631a4088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T02:20:12.271 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66175]: 2024-09-21 02:20:11.882620721 +0000 UTC m=+0.434741394 container start 14794681295d882d6dcc2cc005243dee8a5d0f54b3eba47eb93ab0f6631a4088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T02:20:12.271 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:11 smithi123 podman[66175]: 2024-09-21 02:20:11.979126177 +0000 UTC m=+0.531246852 container attach 14794681295d882d6dcc2cc005243dee8a5d0f54b3eba47eb93ab0f6631a4088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:12.271 INFO:journalctl@ceph.osd.1.smithi123.stdout:Sep 21 02:20:12 smithi123 podman[66175]: 2024-09-21 02:20:12.081608154 +0000 UTC m=+0.633728845 container died 14794681295d882d6dcc2cc005243dee8a5d0f54b3eba47eb93ab0f6631a4088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T02:20:12.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.1.service' 2024-09-21T02:20:12.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:12.732 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T02:20:12.732 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T02:20:12.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.2 2024-09-21T02:20:13.045 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:12 smithi161 systemd[1]: Stopping Ceph osd.2 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:13.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T02:20:13.044+0000 7f42186ad640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:13.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T02:20:13.044+0000 7f42186ad640 -1 osd.2 104 *** Got signal Terminated *** 2024-09-21T02:20:13.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2[39331]: 2024-09-21T02:20:13.044+0000 7f42186ad640 -1 osd.2 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:18.988 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:18 smithi161 podman[68021]: 2024-09-21 02:20:18.685220451 +0000 UTC m=+5.756836045 container died d201fb31ecb2d56d6bf65fb69fa66428669da2a82c82b42973dcdfe3263ea94e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T02:20:19.311 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:19 smithi161 podman[68021]: 2024-09-21 02:20:19.094520748 +0000 UTC m=+6.166136328 container cleanup d201fb31ecb2d56d6bf65fb69fa66428669da2a82c82b42973dcdfe3263ea94e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:20:19.311 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:19 smithi161 bash[68021]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2 2024-09-21T02:20:19.665 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:19 smithi161 podman[68036]: 2024-09-21 02:20:19.329434569 +0000 UTC m=+0.626481502 container remove d201fb31ecb2d56d6bf65fb69fa66428669da2a82c82b42973dcdfe3263ea94e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T02:20:20.066 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:19 smithi161 podman[68135]: 2024-09-21 02:20:19.674391926 +0000 UTC m=+0.015923234 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:20.066 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:19 smithi161 podman[68135]: 2024-09-21 02:20:19.796987263 +0000 UTC m=+0.138518562 container create d83de8d69c32fd315bb24afc3e78229d95e6faabdcb206d14008fbe223a76a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T02:20:20.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:20 smithi161 podman[68135]: 2024-09-21 02:20:20.06618741 +0000 UTC m=+0.407718705 container init d83de8d69c32fd315bb24afc3e78229d95e6faabdcb206d14008fbe223a76a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:20.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:20 smithi161 podman[68135]: 2024-09-21 02:20:20.070184865 +0000 UTC m=+0.411716168 container start d83de8d69c32fd315bb24afc3e78229d95e6faabdcb206d14008fbe223a76a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T02:20:20.415 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:20 smithi161 podman[68135]: 2024-09-21 02:20:20.158049718 +0000 UTC m=+0.499581022 container attach d83de8d69c32fd315bb24afc3e78229d95e6faabdcb206d14008fbe223a76a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:20:20.416 INFO:journalctl@ceph.osd.2.smithi161.stdout:Sep 21 02:20:20 smithi161 podman[68135]: 2024-09-21 02:20:20.303060983 +0000 UTC m=+0.644592291 container died d83de8d69c32fd315bb24afc3e78229d95e6faabdcb206d14008fbe223a76a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:20:20.954 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.2.service' 2024-09-21T02:20:20.993 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:20.993 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T02:20:20.994 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T02:20:20.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.3 2024-09-21T02:20:21.324 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:21 smithi016 systemd[1]: Stopping Ceph osd.3 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:21.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:21 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:21.323+0000 7ff9f1bb1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:21.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:21 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:21.323+0000 7ff9f1bb1640 -1 osd.3 104 *** Got signal Terminated *** 2024-09-21T02:20:21.658 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:21 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:21.323+0000 7ff9f1bb1640 -1 osd.3 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:23.408 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:22 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:22.969+0000 7ff9ea194640 -1 osd.3 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:56.992971+0000 front 2024-09-21T02:19:56.992937+0000 (oldest deadline 2024-09-21T02:20:22.892972+0000) 2024-09-21T02:20:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:22.937+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:24.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:23.895+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:24.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:23.973+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:24.408 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:23 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:23.953+0000 7ff9ea194640 -1 osd.3 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:56.992971+0000 front 2024-09-21T02:19:56.992937+0000 (oldest deadline 2024-09-21T02:20:22.892972+0000) 2024-09-21T02:20:24.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:24.183+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:24.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:24 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:24.415+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:25.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:24.862+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:25.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:24.989+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:25.404 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:24 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:24.937+0000 7ff9ea194640 -1 osd.3 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:56.992971+0000 front 2024-09-21T02:19:56.992937+0000 (oldest deadline 2024-09-21T02:20:22.892972+0000) 2024-09-21T02:20:25.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:25 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:25.155+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:25.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:25.403+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:26.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:25.865+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:26.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:25.983+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:26.402 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:25 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3[54942]: 2024-09-21T02:20:25.967+0000 7ff9ea194640 -1 osd.3 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:56.992971+0000 front 2024-09-21T02:19:56.992937+0000 (oldest deadline 2024-09-21T02:20:22.892972+0000) 2024-09-21T02:20:26.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:26 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:26.205+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:26.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:26 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:26.401+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:26.909 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:26.617+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:27.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:27.006+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:27.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:26.908+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:27.516 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:27 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:27.418+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:27.516 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:27 smithi016 podman[105014]: 2024-09-21 02:20:27.218108189 +0000 UTC m=+6.016403598 container died b110ba99eed9a8d3583acec90e16606dc6b3830d5554cc7e4857002f60740ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:20:27.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:27 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:27.184+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:27.908 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:27 smithi016 podman[105014]: 2024-09-21 02:20:27.690491871 +0000 UTC m=+6.488787275 container cleanup b110ba99eed9a8d3583acec90e16606dc6b3830d5554cc7e4857002f60740ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T02:20:27.908 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:27 smithi016 bash[105014]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3 2024-09-21T02:20:27.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:27.657+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:28.317 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:27 smithi016 podman[105031]: 2024-09-21 02:20:27.958886801 +0000 UTC m=+0.734059787 container remove b110ba99eed9a8d3583acec90e16606dc6b3830d5554cc7e4857002f60740ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-21T02:20:28.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:27.994+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:28.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:27.918+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:28.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:28 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:28.227+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:28.648 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:28 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:28.437+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:28.648 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.2203062 +0000 UTC m=+0.016814788 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:28.648 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.318053018 +0000 UTC m=+0.114561588 container create 98002073fee74002c38dd99e7d74c558e37c945b9e887808090c0873f26b6b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T02:20:28.648 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.556516596 +0000 UTC m=+0.353025180 container init 98002073fee74002c38dd99e7d74c558e37c945b9e887808090c0873f26b6b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:20:28.648 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.559867993 +0000 UTC m=+0.356376562 container start 98002073fee74002c38dd99e7d74c558e37c945b9e887808090c0873f26b6b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T02:20:28.649 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.64814873 +0000 UTC m=+0.444657300 container attach 98002073fee74002c38dd99e7d74c558e37c945b9e887808090c0873f26b6b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:28.871 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:28.610+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:28.908 INFO:journalctl@ceph.osd.3.smithi016.stdout:Sep 21 02:20:28 smithi016 podman[105119]: 2024-09-21 02:20:28.75468094 +0000 UTC m=+0.551189517 container died 98002073fee74002c38dd99e7d74c558e37c945b9e887808090c0873f26b6b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True) 2024-09-21T02:20:29.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:28.994+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:29.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:28.871+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:29.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.3.service' 2024-09-21T02:20:29.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:29.451 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T02:20:29.452 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T02:20:29.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.4 2024-09-21T02:20:29.479 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:29 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:29.197+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:29.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:29 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:29.465+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:29.770 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:29 smithi123 systemd[1]: Stopping Ceph osd.4 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:29.853 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:29.571+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:30.150 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:29 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:29.793+0000 7f58a35d5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:30.151 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:29 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:29.793+0000 7f58a35d5640 -1 osd.4 104 *** Got signal Terminated *** 2024-09-21T02:20:30.151 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:29 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:29.793+0000 7f58a35d5640 -1 osd.4 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:30.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:29.992+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:30.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:29.852+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:30.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:30 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:30.149+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:30.823 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:30.546+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:30.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:30 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:30.429+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:31.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:30.822+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:31.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:31.022+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:31.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:31 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:31.175+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:31.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:31 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:31.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:31.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:31.582+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:31.915 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:31.813+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:32.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:31.977+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:32.520 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:32 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:32.398+0000 7f589fbc0640 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.216679+0000 front 2024-09-21T02:20:09.216705+0000 (oldest deadline 2024-09-21T02:20:31.516936+0000) 2024-09-21T02:20:32.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:32 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:32.209+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:32.832 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:32.576+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:32.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:32 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:32.482+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:33.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:33 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:33.025+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:33.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:32.831+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:33.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:32.831+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:33.520 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:33 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:33.226+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:33.521 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:33 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:33.360+0000 7f589fbc0640 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.216679+0000 front 2024-09-21T02:20:09.216705+0000 (oldest deadline 2024-09-21T02:20:31.516936+0000) 2024-09-21T02:20:33.790 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:33 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:33.525+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:33.790 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:33 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:33.525+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:33.864 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:33 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:33.539+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:33.864 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:33 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:33.539+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:34.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:33 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:33.789+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:34.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:34 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:34.013+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:34.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:33 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:33.863+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:34.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:33 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:33.863+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:34.386 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:34 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:34.258+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:34.387 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:34 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:34.258+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:34.387 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:34 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:34.132+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:34.770 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:34 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4[44251]: 2024-09-21T02:20:34.385+0000 7f589fbc0640 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.216679+0000 front 2024-09-21T02:20:09.216705+0000 (oldest deadline 2024-09-21T02:20:31.516936+0000) 2024-09-21T02:20:34.788 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:34 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:34.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:34.788 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:34 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:34.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:34.815 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:34 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:34.494+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:34.815 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:34 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:34.494+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:35.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:34 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:34.788+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:35.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:35 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:35.028+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:35.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:34 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:34.815+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:35.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:34 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:34.815+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:35.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:35 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:35.168+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:35.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:35 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:35.276+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:35.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:35 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:35.276+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:35.780 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:35.469+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:35.780 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:35.469+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:35.786 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:35 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:35.467+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:35.786 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:35 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:35.467+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:35.945 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:35 smithi123 podman[66289]: 2024-09-21 02:20:35.653210904 +0000 UTC m=+5.996945195 container died a359f849c2e29407f678a3939398e6b95895024ed109f1b12c2cc55a526d950d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:36.044 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:35 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:35.785+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:36.044 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:35 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:35.785+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:36.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:35.779+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:36.270 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:36 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:36.137+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:36.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66289]: 2024-09-21 02:20:36.035745633 +0000 UTC m=+6.379479966 container cleanup a359f849c2e29407f678a3939398e6b95895024ed109f1b12c2cc55a526d950d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T02:20:36.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 bash[66289]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4 2024-09-21T02:20:36.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:36 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:36.043+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:36.643 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66306]: 2024-09-21 02:20:36.303695035 +0000 UTC m=+0.651785695 container remove a359f849c2e29407f678a3939398e6b95895024ed109f1b12c2cc55a526d950d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T02:20:36.644 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:36 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:36.303+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:36.644 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:36 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:36.303+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:36.752 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:36 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:36.459+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:36.752 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:36 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:36.459+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:36.831 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:36 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:36.453+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:36.831 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:36 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:36.453+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:36.974 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66398]: 2024-09-21 02:20:36.546022706 +0000 UTC m=+0.015622636 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:36.974 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66398]: 2024-09-21 02:20:36.670140441 +0000 UTC m=+0.139740361 container create aac07a754b6c1bae98289ae3898343b05018e8ee1a7c54f2882ef8d25156592c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2) 2024-09-21T02:20:37.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:36 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:36.751+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:37.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:37 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:37.024+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:37.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:36 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:36.830+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:37.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:36 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:36.830+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:37.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66398]: 2024-09-21 02:20:36.973677043 +0000 UTC m=+0.443276970 container init aac07a754b6c1bae98289ae3898343b05018e8ee1a7c54f2882ef8d25156592c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T02:20:37.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:36 smithi123 podman[66398]: 2024-09-21 02:20:36.977234643 +0000 UTC m=+0.446834570 container start aac07a754b6c1bae98289ae3898343b05018e8ee1a7c54f2882ef8d25156592c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:20:37.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:37 smithi123 podman[66398]: 2024-09-21 02:20:37.090786755 +0000 UTC m=+0.560386692 container attach aac07a754b6c1bae98289ae3898343b05018e8ee1a7c54f2882ef8d25156592c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T02:20:37.271 INFO:journalctl@ceph.osd.4.smithi123.stdout:Sep 21 02:20:37 smithi123 podman[66398]: 2024-09-21 02:20:37.175142146 +0000 UTC m=+0.644742068 container died aac07a754b6c1bae98289ae3898343b05018e8ee1a7c54f2882ef8d25156592c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:20:37.271 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:37 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:37.158+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:37.741 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:37 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:37.454+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:37.741 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:37 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:37.454+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:37.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:37 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:37.340+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:37.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:37 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:37.340+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:37.827 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:37 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:37.431+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:37.827 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:37 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:37.431+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:37.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T02:20:37.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:37.929 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T02:20:37.930 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T02:20:37.930 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.5 2024-09-21T02:20:38.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:37 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:37.740+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:38.165 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:37 smithi161 systemd[1]: Stopping Ceph osd.5 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:38.166 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:38.062+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:38.166 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:37 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:37.827+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:38.166 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:37 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:37.827+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:38.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:38 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:38.190+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:38.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:38 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:38.384+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:38.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:38 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:38.384+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:38.665 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:38.239+0000 7fe75c3f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:38.665 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:38.239+0000 7fe75c3f2640 -1 osd.5 104 *** Got signal Terminated *** 2024-09-21T02:20:38.666 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:38.239+0000 7fe75c3f2640 -1 osd.5 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:38.666 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:38.446+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:38.666 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:38.446+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:38.722 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:38 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:38.446+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:38.722 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:38 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:38.446+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:39.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:38 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:38.721+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:39.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:39.039+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:39.166 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:38.819+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:39.166 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:38 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:38.819+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:39.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:39.197+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:39.521 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:39.197+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:39.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:39.394+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:39.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:39 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:39.394+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:39.665 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:39.406+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:39.665 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:39.406+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:39.743 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:39 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:39.429+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:39.743 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:39 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:39.429+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:40.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:39 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:39.743+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:40.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:39 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:39.743+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:40.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:40.049+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:40.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:39.857+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:40.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:39.857+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:40.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:39 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:39.857+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:40.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:40.212+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:40.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:40.212+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:40.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:40.430+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:40.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:40.430+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:40.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:40 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:40.430+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:40.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:40 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:40.403+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:40.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:40 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:40.403+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:40.887 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:40.449+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:40.888 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:40.449+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:41.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:40 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:40.776+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:41.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:40 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:40.776+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:41.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:41.089+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:41.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:40.887+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:41.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:40.887+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:41.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:40 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:40.887+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:41.521 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:41.249+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:41.521 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:41.249+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:41.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:41.388+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:41.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:41.388+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:41.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:41 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:41.388+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:41.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:41 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:41.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:41.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:41 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:41.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:41.909 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:41.487+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:41.909 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:41.487+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:42.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:41 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:41.750+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:42.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:41 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:41.750+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:42.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:42.104+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:42.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:41.909+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:42.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:41.909+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:42.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:41 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:41.909+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:42.726 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:42 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:42.412+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:42.726 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:42 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:42.412+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:42.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:42 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:42.296+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:42.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:42 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:42.296+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:42.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:42 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:42.375+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:42.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:42 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:42.375+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:42.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:42 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:42.375+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:42.889 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:42.522+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:57.075540+0000 front 2024-09-21T02:19:57.075568+0000 (oldest deadline 2024-09-21T02:20:25.876102+0000) 2024-09-21T02:20:42.889 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:42.522+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.377332+0000 front 2024-09-21T02:20:09.377297+0000 (oldest deadline 2024-09-21T02:20:33.477198+0000) 2024-09-21T02:20:42.889 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5[45842]: 2024-09-21T02:20:42.522+0000 7fe7549d5640 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.378131+0000 front 2024-09-21T02:20:16.378155+0000 (oldest deadline 2024-09-21T02:20:41.678284+0000) 2024-09-21T02:20:43.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:42 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:42.725+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:43.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:42 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:42.725+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:43.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:43 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:43.131+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:43.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:42.888+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:43.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:42.888+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:43.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:42 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:42.888+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:43.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:43 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:43.257+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:43.521 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:43 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:43.257+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:43.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:43 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:43.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:43.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:43 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:43.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:43.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:43 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:43.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:43.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:43 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:43.364+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:43.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:43 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:43.364+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:43.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:43 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:43.364+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:44.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:43 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:43.692+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:44.158 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:43 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:43.692+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:44.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:44.096+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:44.165 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:44.096+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:44.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:43 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:43.883+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:44.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:43 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:43.883+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:44.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:43 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:43.883+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:44.532 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:44 smithi161 podman[68239]: 2024-09-21 02:20:44.260890262 +0000 UTC m=+6.139566357 container died 1ce949d811d989a8465135592f724f804197a4fd88775df8d5e6fde0cb23c5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T02:20:44.655 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:44.352+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:44.655 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:44.352+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:44.655 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:44.352+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:44.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:44 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:44.300+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:44.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:44 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:44.300+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:44.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:44 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:44.309+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:44.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:44 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:44.309+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:44.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:44 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:44.309+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:44.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:44.654+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:44.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:44.654+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:44.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:44 smithi161 podman[68239]: 2024-09-21 02:20:44.639317051 +0000 UTC m=+6.517993130 container cleanup 1ce949d811d989a8465135592f724f804197a4fd88775df8d5e6fde0cb23c5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T02:20:44.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:44 smithi161 bash[68239]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5 2024-09-21T02:20:45.294 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:44 smithi161 podman[68254]: 2024-09-21 02:20:44.957364522 +0000 UTC m=+0.694978785 container remove 1ce949d811d989a8465135592f724f804197a4fd88775df8d5e6fde0cb23c5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T02:20:45.295 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:45 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:45.050+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:45.295 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:45 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:45.050+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:45.295 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:44.924+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:45.295 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:44.924+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:45.295 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:44 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:44.924+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:45.577 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.19725197 +0000 UTC m=+0.016120288 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:45.578 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.333365336 +0000 UTC m=+0.152233682 container create f255b414a765527586f257bf44513bac630a1766f51cbe76816c22ad8564e952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T02:20:45.578 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.577383402 +0000 UTC m=+0.396251717 container init f255b414a765527586f257bf44513bac630a1766f51cbe76816c22ad8564e952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T02:20:45.629 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:45 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:45.354+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:45.629 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:45 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:45.354+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:45.629 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:45 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:45.354+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:45.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:45 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:45.289+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:45.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:45 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:45.289+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:45.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:45 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:45.359+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:45.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:45 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:45.359+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:45.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:45 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:45.359+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:45.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:45 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:45.628+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:45.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:45 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:45.628+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:45.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.580817768 +0000 UTC m=+0.399686081 container start f255b414a765527586f257bf44513bac630a1766f51cbe76816c22ad8564e952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T02:20:45.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.677406947 +0000 UTC m=+0.496275268 container attach f255b414a765527586f257bf44513bac630a1766f51cbe76816c22ad8564e952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T02:20:45.915 INFO:journalctl@ceph.osd.5.smithi161.stdout:Sep 21 02:20:45 smithi161 podman[68347]: 2024-09-21 02:20:45.772004047 +0000 UTC m=+0.590872360 container died f255b414a765527586f257bf44513bac630a1766f51cbe76816c22ad8564e952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:20:46.398 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:46 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:46.059+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:46.398 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:46 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:46.059+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:46.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:45 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:45.957+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:46.398 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:45 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:45.957+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:46.399 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:45 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:45.957+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:46.424 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T02:20:46.464 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:46.465 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T02:20:46.465 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T02:20:46.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.6 2024-09-21T02:20:46.520 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:46 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:46.244+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:46.521 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:46 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:46.244+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:46.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:46 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:46.330+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:46.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:46 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:46.330+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:46.521 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:46 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:46.330+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:46.646 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 systemd[1]: Stopping Ceph osd.6 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:46.646 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:46.351+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:46.646 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:46.351+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:46.646 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:46.351+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:46.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:46.644+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:46.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:46.644+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:46.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:46.813+0000 7f2a82f70640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:46.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:46.813+0000 7f2a82f70640 -1 osd.6 104 *** Got signal Terminated *** 2024-09-21T02:20:46.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:46.813+0000 7f2a82f70640 -1 osd.6 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:47.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:47 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:47.026+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:47.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:47 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:47.026+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:47.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:46 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:46.974+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:47.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:46 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:46.974+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:47.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:46 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:46.974+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:47.658 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:47.622+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:47.658 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:47.622+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:47.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:47.389+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:47.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:47.389+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:47.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:47 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:47.389+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:47.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:47 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:47.291+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:47.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:47 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:47.291+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:47.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:47 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:47.306+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:47.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:47 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:47.306+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:47.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:47 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:47.306+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:48.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:47 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:47.994+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:48.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:47 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:47.994+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:48.318 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:47 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:47.994+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:48.318 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:48.032+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:48.318 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:48.032+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:48.318 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:48.032+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:48.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:48 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:48.326+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:48.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:48 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:48.326+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:48.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:48 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:48.311+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:48.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:48 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:48.311+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:48.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:48 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:48.311+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:48.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:48 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:48.584+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:48.909 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:48 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:48.584+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:48.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:48 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:48.433+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:48.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:48 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:48.433+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:48.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:48 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:48.433+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:49.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:49.079+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:49.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:49.079+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:49.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:49.079+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:49.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:48.990+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:49.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:48.990+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:49.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:48 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:48.990+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:49.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:49 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:49.367+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:49.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:49 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:49.367+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:49.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:49 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:49.305+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:49.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:49 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:49.305+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:49.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:49 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:49.305+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:49.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:49.593+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:49.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:49.593+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:49.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:49.456+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:49.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:49.456+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:49.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:49.456+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:49.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:49 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:49.456+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:50.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:49.986+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:50.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:49.986+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:50.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:49 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:49.986+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:50.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:50 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:50.051+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:50.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:50 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:50.051+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:50.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:50 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:50.051+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:50.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:50 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:50.320+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:50.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:50 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:50.320+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:50.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:50 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:50.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:50.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:50 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:50.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:50.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:50 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:50.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:50.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:50.578+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:50.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:50.578+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:50.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:50.494+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:50.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:50.494+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:50.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:50.494+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:50.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:50 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:50.494+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:51.078+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:51.078+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:51.078+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:51.035+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:51.035+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:51.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:51 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:51.035+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:51.363+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:51.363+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:51.363+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:51.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:51.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:51.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:51.338+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:51.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:51.553+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.530505+0000 front 2024-09-21T02:20:09.530424+0000 (oldest deadline 2024-09-21T02:20:33.630517+0000) 2024-09-21T02:20:51.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:51.553+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.631099+0000 front 2024-09-21T02:20:13.631173+0000 (oldest deadline 2024-09-21T02:20:39.531273+0000) 2024-09-21T02:20:51.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6[61537]: 2024-09-21T02:20:51.553+0000 7f2a7b553640 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.932663+0000 front 2024-09-21T02:20:25.932761+0000 (oldest deadline 2024-09-21T02:20:51.232909+0000) 2024-09-21T02:20:51.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:51.476+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:51.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:51.476+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:51.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:51.476+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:51.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:51 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:51.476+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:52.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:52.084+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:52.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:52.084+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:52.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:52.084+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:52.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:52.039+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:52.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:52.039+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:52.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:52 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:52.039+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:52.345+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:52.345+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:52.345+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:52.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:52.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:52.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:52.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:52 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:52.349+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:52.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:52 smithi016 podman[105221]: 2024-09-21 02:20:52.728842995 +0000 UTC m=+6.062380422 container died 1d5195f746a4231d484570894cde4fb63cdf7ee26403d6fab7d8253a8c0c4fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T02:20:52.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:52 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:52.499+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:52.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:52 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:52.499+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:52.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:52 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:52.499+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:52.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:52 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:52.499+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:53.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:53 smithi016 podman[105221]: 2024-09-21 02:20:53.243138657 +0000 UTC m=+6.576676136 container cleanup 1d5195f746a4231d484570894cde4fb63cdf7ee26403d6fab7d8253a8c0c4fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T02:20:53.408 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:53 smithi016 bash[105221]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6 2024-09-21T02:20:53.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:53.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:53.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:53.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:53.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:53.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:53.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:53.087+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:53.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:53.087+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:53.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:53 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:53.087+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:53.672 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:53.337+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:53.672 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:53.337+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:53.672 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:53.337+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:53.672 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:53.362+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:53.673 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:53.362+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:53.673 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:53.362+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:53.673 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:53 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:53.362+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:53.908 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:53 smithi016 podman[105247]: 2024-09-21 02:20:53.498859411 +0000 UTC m=+0.763666982 container remove 1d5195f746a4231d484570894cde4fb63cdf7ee26403d6fab7d8253a8c0c4fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:20:53.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:53.487+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:53.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:53.487+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:53.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:53.487+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:53.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:53 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:53.487+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:54.177 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:53 smithi016 podman[105338]: 2024-09-21 02:20:53.818497597 +0000 UTC m=+0.016544833 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:20:54.178 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:53 smithi016 podman[105338]: 2024-09-21 02:20:53.924956951 +0000 UTC m=+0.123004209 container create abd2241b9bf349f1248259c00aa31890ba3e2ccb2979a1da03687d2693fab77b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T02:20:54.178 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:54 smithi016 podman[105338]: 2024-09-21 02:20:54.177604238 +0000 UTC m=+0.375651474 container init abd2241b9bf349f1248259c00aa31890ba3e2ccb2979a1da03687d2693fab77b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T02:20:54.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:54.100+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:54.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:54.100+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:54.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:54.100+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:54.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:54.064+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:54.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:54.064+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:54.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:54 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:54.064+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:54.473 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:54 smithi016 podman[105338]: 2024-09-21 02:20:54.18188434 +0000 UTC m=+0.379931564 container start abd2241b9bf349f1248259c00aa31890ba3e2ccb2979a1da03687d2693fab77b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T02:20:54.473 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:54 smithi016 podman[105338]: 2024-09-21 02:20:54.30287866 +0000 UTC m=+0.500925890 container attach abd2241b9bf349f1248259c00aa31890ba3e2ccb2979a1da03687d2693fab77b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T02:20:54.473 INFO:journalctl@ceph.osd.6.smithi016.stdout:Sep 21 02:20:54 smithi016 podman[105338]: 2024-09-21 02:20:54.374773734 +0000 UTC m=+0.572820966 container died abd2241b9bf349f1248259c00aa31890ba3e2ccb2979a1da03687d2693fab77b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:54.344+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:54.344+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:54.344+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:54.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:54.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:54.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:54.688 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:54.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:54.731 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:54 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:54.472+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:54.731 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:54 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:54.472+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:54.732 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:54 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:54.472+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:54.732 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:54 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:54.472+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:55.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T02:20:55.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:20:55.094 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T02:20:55.094 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T02:20:55.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.7 2024-09-21T02:20:55.270 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 systemd[1]: Stopping Ceph osd.7 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:20:55.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:55.101+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:55.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:55.101+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:55.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:55.101+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:55.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:55.033+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:55.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:55.033+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:55.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:55 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:55.033+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:55.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:55.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:55.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:55.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:55.371+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.356+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.356+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.356+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.387+0000 7f7499c7d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.387+0000 7f7499c7d640 -1 osd.7 104 *** Got signal Terminated *** 2024-09-21T02:20:55.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:55 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:55.387+0000 7f7499c7d640 -1 osd.7 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:20:55.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:55 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:55.428+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:55.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:55 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:55.428+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:55.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:55 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:55.428+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:55.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:55 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:55.428+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:56.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:56.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:56.067+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:56.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:56.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:56.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:56 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:56.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:56.385+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:56.385+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:56.385+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:56.377+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:56.377+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:56.377+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:56.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:56 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:56.377+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:56.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:56.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:56.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:56.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:56.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:56.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:56.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:56 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:56.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:57.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:57.116+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:57.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:57.116+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:57.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:57.116+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:57.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:57.116+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:20:57.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:57.045+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:57.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:57.045+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:57.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:57.045+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:57.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:57 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:57.045+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:57.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:57.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:57.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:57.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:34.456196+0000 front 2024-09-21T02:20:34.456236+0000 (oldest deadline 2024-09-21T02:20:56.756287+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:57.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:57.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:57.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:57.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:57 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:57.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:57.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:57 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:57.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:57.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:57 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:57.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:57.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:57 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:57.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:57.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:57 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:57.481+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:58.147+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:58.147+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:58.147+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:58.147+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:58.083+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:58.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:58.083+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:58.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:58.083+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:58.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:58 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:58.083+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:20:58.770 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:58.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:58.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:58.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:58.411+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:34.456196+0000 front 2024-09-21T02:20:34.456236+0000 (oldest deadline 2024-09-21T02:20:56.756287+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:58.428+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:58.428+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:58.428+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:58.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:58.428+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:58.911 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:58.502+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:58.911 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:58.502+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:58.911 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:58.502+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:58.911 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:58.502+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:58.911 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:58 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:58.502+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:20:59.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:59.158+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:20:59.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:59.158+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:20:59.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:59.158+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:20:59.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:20:59.158+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:20:59.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:59.055+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:20:59.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:59.055+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:20:59.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:59.055+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:20:59.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:20:59 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:20:59.055+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:59.435+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:08.049819+0000 front 2024-09-21T02:20:08.049861+0000 (oldest deadline 2024-09-21T02:20:33.350024+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:59.435+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:16.151449+0000 front 2024-09-21T02:20:16.151496+0000 (oldest deadline 2024-09-21T02:20:38.952207+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:59.435+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:21.754006+0000 front 2024-09-21T02:20:21.753807+0000 (oldest deadline 2024-09-21T02:20:50.554428+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7[50559]: 2024-09-21T02:20:59.435+0000 7f7492260640 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:34.456196+0000 front 2024-09-21T02:20:34.456236+0000 (oldest deadline 2024-09-21T02:20:56.756287+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:59.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:59.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:59.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:20:59.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:20:59 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:20:59.415+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:20:59.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:59.497+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:20:59.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:59.497+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:20:59.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:59.497+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:20:59.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:59.497+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:20:59.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:20:59 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:20:59.497+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:00.097+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:00.097+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:00.097+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:00.097+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:00.199+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:00.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:00.199+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:00.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:00.199+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:00.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:00 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:00.199+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:00.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:00.446+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:00.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:00.446+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:00.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:00.446+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:00.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:00 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:00.446+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:00.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:00.511+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:00.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:00.511+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:00.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:00.511+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:00.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:00.511+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:00.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:00 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:00.511+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:01.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:01.068+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:01.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:01.068+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:01.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:01.068+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:01.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:01.068+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:01.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:01.167+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:01.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:01.167+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:01.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:01.167+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:01.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:01 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:01.167+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:01.455 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:01 smithi123 podman[66508]: 2024-09-21 02:21:01.171552738 +0000 UTC m=+5.880738135 container died 3f82520d5dcbad9496a5d17b8c113820c7fd2e3291c70ed39a473026522188c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:21:01.765 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:01 smithi123 podman[66508]: 2024-09-21 02:21:01.554652009 +0000 UTC m=+6.263837400 container cleanup 3f82520d5dcbad9496a5d17b8c113820c7fd2e3291c70ed39a473026522188c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T02:21:01.765 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:01 smithi123 bash[66508]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7 2024-09-21T02:21:01.765 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:01 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:01.490+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:01.765 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:01 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:01.490+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:01.765 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:01 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:01.490+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:01.766 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:01 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:01.490+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:01.766 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:01 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:01.490+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:01.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:01 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:01.488+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:01.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:01 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:01.488+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:01.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:01 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:01.488+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:01.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:01 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:01.488+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:01.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:01 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:01.488+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:02.020 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:01 smithi123 podman[66528]: 2024-09-21 02:21:01.764222797 +0000 UTC m=+0.584799633 container remove 3f82520d5dcbad9496a5d17b8c113820c7fd2e3291c70ed39a473026522188c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:21:02.367 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:01.984050921 +0000 UTC m=+0.015986152 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:21:02.368 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:02.106859206 +0000 UTC m=+0.138794483 container create 06f0927959c1e91fd7faca948b380570555129ee0ed5ee4e1a6b39a303bbd67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T02:21:02.368 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:02.367414107 +0000 UTC m=+0.399349353 container init 06f0927959c1e91fd7faca948b380570555129ee0ed5ee4e1a6b39a303bbd67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T02:21:02.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:02.200+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:02.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:02.200+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:02.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:02.200+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:02.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:02.200+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:02.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:02.047+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:02.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:02.048+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:02.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:02.048+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:02.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:02 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:02.048+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:02.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:02.500+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:02.500+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:02.500+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:02.500+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:02.500+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:02.370929161 +0000 UTC m=+0.402864399 container start 06f0927959c1e91fd7faca948b380570555129ee0ed5ee4e1a6b39a303bbd67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:02.450682558 +0000 UTC m=+0.482617811 container attach 06f0927959c1e91fd7faca948b380570555129ee0ed5ee4e1a6b39a303bbd67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:21:02.771 INFO:journalctl@ceph.osd.7.smithi123.stdout:Sep 21 02:21:02 smithi123 podman[66614]: 2024-09-21 02:21:02.559599861 +0000 UTC m=+0.591535104 container died 06f0927959c1e91fd7faca948b380570555129ee0ed5ee4e1a6b39a303bbd67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T02:21:02.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:02 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:02.448+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:02.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:02 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:02.448+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:02.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:02 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:02.448+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:02.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:02 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:02.448+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:02.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:02 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:02.448+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:03.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T02:21:03.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:03.248 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T02:21:03.248 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-21T02:21:03.248 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.8 2024-09-21T02:21:03.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:03.073+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:03.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:03.073+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:03.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:03.073+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:03.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:03.073+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:03.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.231+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:03.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.231+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:03.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.231+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:03.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.231+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:03.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 systemd[1]: Stopping Ceph osd.8 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:03.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:03 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:03.498+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:03.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:03 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:03.498+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:03.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:03 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:03.498+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:03.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:03 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:03.498+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:03.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:03 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:03.498+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:03.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:03 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:03.492+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:03.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:03 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:03.492+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:03.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:03 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:03.492+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:03.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:03 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:03.492+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:03.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:03 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:03.492+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:03.915 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.566+0000 7ff3ffb33640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:03.915 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.566+0000 7ff3ffb33640 -1 osd.8 104 *** Got signal Terminated *** 2024-09-21T02:21:03.915 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:03 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:03.566+0000 7ff3ffb33640 -1 osd.8 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:04.193+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:04.193+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:04.193+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:04.193+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.024+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:04.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.024+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:04.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.024+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:04.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.024+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:04.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:04.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:04.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:04.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:04.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:04.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:04.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:04.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:04.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:04 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:04.443+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:04.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:04.503+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:04.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:04.503+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:04.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:04.503+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:04.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:04.503+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:04.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:04 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:04.503+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:05.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:05.191+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:05.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:05.191+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:05.191+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:05.191+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:05 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:05.191+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:39.544947+0000 front 2024-09-21T02:20:39.545029+0000 (oldest deadline 2024-09-21T02:21:04.844983+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.981+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.981+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.981+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.981+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:05.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:04 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:04.981+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:05.770 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:05.549+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:05.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:05.549+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:05.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:05.549+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:05.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:05.549+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:05.771 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:05 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:05.549+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:05.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:05 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:05.422+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:05.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:05 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:05.422+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:05.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:05 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:05.422+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:05.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:05 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:05.422+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:05.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:05 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:05.422+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:06.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:06.236+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:06.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:06.236+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:06.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:06.236+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:06.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:06.236+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:06.236+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:39.544947+0000 front 2024-09-21T02:20:39.545029+0000 (oldest deadline 2024-09-21T02:21:04.844983+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:06.025+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:06.025+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:06.025+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:06.025+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:06.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:06.025+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:06.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:06 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:06.390+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:06.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:06 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:06.390+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:06.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:06 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:06.390+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:06.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:06 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:06.390+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:06.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:06 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:06.390+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:07.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:06 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:06.568+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:07.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:07.264+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:07.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:07.264+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:07.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:07.264+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:07.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:07.264+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:07.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:07.264+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:39.544947+0000 front 2024-09-21T02:20:39.545029+0000 (oldest deadline 2024-09-21T02:21:04.844983+0000) 2024-09-21T02:21:07.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:07.067+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:07.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:07.067+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:07.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:07.067+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:07.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:07.067+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:07.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:07 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:07.067+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:07.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:07 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:07.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:07.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:07 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:07.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:07.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:07 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:07.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:07.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:07 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:07.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:07.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:07 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:07.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:08.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:08.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:08.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:08.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:08.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:08.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:07 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:07.604+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:08.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:08.103+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:08.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:08.103+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:08.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:08.103+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:08.103+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:08.103+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:08.301+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:20:00.137642+0000 front 2024-09-21T02:20:00.137820+0000 (oldest deadline 2024-09-21T02:20:23.637845+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:08.301+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:15.641464+0000 front 2024-09-21T02:20:15.641445+0000 (oldest deadline 2024-09-21T02:20:43.242235+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:08.301+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.543060+0000 front 2024-09-21T02:20:25.542923+0000 (oldest deadline 2024-09-21T02:20:47.243074+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:08.301+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.743812+0000 front 2024-09-21T02:20:30.743883+0000 (oldest deadline 2024-09-21T02:20:56.643965+0000) 2024-09-21T02:21:08.416 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:08 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8[52163]: 2024-09-21T02:21:08.301+0000 7ff3fc11e640 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:39.544947+0000 front 2024-09-21T02:20:39.545029+0000 (oldest deadline 2024-09-21T02:21:04.844983+0000) 2024-09-21T02:21:08.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:08.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:08.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:08.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:08.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:08.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:08 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:08.409+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:09.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:08 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:08.587+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:09.415 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:09 smithi161 podman[68452]: 2024-09-21 02:21:09.161962727 +0000 UTC m=+5.716412345 container died 9561e3cc153784524987f281cadf9cc845680304eb737df38d3f052e7a206cce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:21:09.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:09 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:09.150+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:09.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:09 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:09.150+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:09.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:09 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:09.150+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:09.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:09 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:09.150+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:09.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:09 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:09.150+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:09.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:09.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:09.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:09.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:09.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:09.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:09 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:09.391+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:09.783 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:09 smithi161 podman[68452]: 2024-09-21 02:21:09.565782643 +0000 UTC m=+6.120232265 container cleanup 9561e3cc153784524987f281cadf9cc845680304eb737df38d3f052e7a206cce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:21:09.784 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:09 smithi161 bash[68452]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:10.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:09 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:09.613+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:10.111 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:09 smithi161 podman[68468]: 2024-09-21 02:21:09.783565713 +0000 UTC m=+0.620299577 container remove 9561e3cc153784524987f281cadf9cc845680304eb737df38d3f052e7a206cce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:10.362 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:10.169+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:10.362 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:10.169+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:10.362 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:10.169+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:10.362 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:10.169+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:10.362 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:10.169+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:10.363 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.013593179 +0000 UTC m=+0.015405707 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:21:10.363 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.117757117 +0000 UTC m=+0.119569639 container create 04347e967e058b1def8e556667cc5954f7d3e2ae72c5a15195c5343fb3cb2584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T02:21:10.363 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.361950538 +0000 UTC m=+0.363763051 container init 04347e967e058b1def8e556667cc5954f7d3e2ae72c5a15195c5343fb3cb2584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:10.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:10.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:10.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:10.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:10.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:10.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:10 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:10.348+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:10.665 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.365604628 +0000 UTC m=+0.367417142 container start 04347e967e058b1def8e556667cc5954f7d3e2ae72c5a15195c5343fb3cb2584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T02:21:10.665 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.436935552 +0000 UTC m=+0.438748065 container attach 04347e967e058b1def8e556667cc5954f7d3e2ae72c5a15195c5343fb3cb2584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-21T02:21:10.665 INFO:journalctl@ceph.osd.8.smithi161.stdout:Sep 21 02:21:10 smithi161 podman[68555]: 2024-09-21 02:21:10.551475039 +0000 UTC m=+0.553287554 container died 04347e967e058b1def8e556667cc5954f7d3e2ae72c5a15195c5343fb3cb2584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-8-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T02:21:11.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:11.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:11.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:11.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:11.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:11.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:10 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:10.610+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:11.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.8.service' 2024-09-21T02:21:11.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:11.204 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-21T02:21:11.204 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-21T02:21:11.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.9 2024-09-21T02:21:11.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:11 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:11.166+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:11.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:11 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:11.166+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:11.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:11 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:11.166+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:11.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:11 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:11.166+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:11.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:11 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:11.166+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 systemd[1]: Stopping Ceph osd.9 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.396+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.533+0000 7fae77a15640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.533+0000 7fae77a15640 -1 osd.9 104 *** Got signal Terminated *** 2024-09-21T02:21:11.535 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:11 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:11.533+0000 7fae77a15640 -1 osd.9 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:21:12.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:12.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:12.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:12.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:12.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:12.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:11 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:11.625+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:12.376 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:12 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:12.119+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:12.376 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:12 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:12.119+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:12.376 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:12 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:12.119+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:12.376 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:12 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:12.119+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:12.376 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:12 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:12.119+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:12.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:12.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:12.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:12.658 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:12.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:12.659 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:12 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:12.380+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:13.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:13.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:13.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:13.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:13.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:13.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:12 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:12.643+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:13.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:13.132+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:13.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:13.132+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:13.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:13.132+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:13.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:13.132+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:13.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:13 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:13.132+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:13.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:13.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:13.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:13.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:13.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:13.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:13 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:13.420+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:14.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:14.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:14.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:14.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:14.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:14.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:13 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:13.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:14.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:14.120+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:14.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:14.120+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:14.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:14.120+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:14.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:14.120+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:14.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:14.120+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:14.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:14.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:14.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:14.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:14.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:14.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:14 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:14.466+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:15.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:15.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:15.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:15.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:15.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:15.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:14 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:14.656+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:15.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:15.090+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:15.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:15.090+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:15.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:15.090+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:15.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:15.090+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:15.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:15 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:15.090+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:15.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:15.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:15.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:15.908 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:15.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:15.909 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:15 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:15.496+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:16.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:16.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:16.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:16.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:16.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:16.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:15 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:15.630+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:16.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:16 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:16.075+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:16.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:16 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:16.075+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:16.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:16 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:16.075+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:16.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:16 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:16.075+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:16.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:16 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:16.075+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:55.648624+0000 front 2024-09-21T02:19:55.648671+0000 (oldest deadline 2024-09-21T02:20:23.849484+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:07.350871+0000 front 2024-09-21T02:20:07.350689+0000 (oldest deadline 2024-09-21T02:20:32.650728+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:17.252199+0000 front 2024-09-21T02:20:17.252254+0000 (oldest deadline 2024-09-21T02:20:43.052816+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:23.053177+0000 front 2024-09-21T02:20:23.053216+0000 (oldest deadline 2024-09-21T02:20:48.953207+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:28.953672+0000 front 2024-09-21T02:20:28.953806+0000 (oldest deadline 2024-09-21T02:20:58.354411+0000) 2024-09-21T02:21:16.810 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:16 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9[68051]: 2024-09-21T02:21:16.513+0000 7fae74000640 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:42.455876+0000 front 2024-09-21T02:20:42.455693+0000 (oldest deadline 2024-09-21T02:21:07.755522+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:17.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:16 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:16.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:17.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:17 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:17.071+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:17.819 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:17 smithi016 podman[105440]: 2024-09-21 02:21:17.558414795 +0000 UTC m=+6.147376526 container died 86da2410ffb3d9293ded5c8100aa58dd2bd3f7795abd3bcd688f49b5c9318975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T02:21:18.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:18.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:17 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:17.595+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:18.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:17 smithi016 podman[105440]: 2024-09-21 02:21:17.909044471 +0000 UTC m=+6.498006191 container cleanup 86da2410ffb3d9293ded5c8100aa58dd2bd3f7795abd3bcd688f49b5c9318975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:21:18.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:17 smithi016 bash[105440]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9 2024-09-21T02:21:18.316 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:18.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:18.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:18.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:18.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:18.317 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:18.049+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:18.533 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105456]: 2024-09-21 02:21:18.193749499 +0000 UTC m=+0.631665055 container remove 86da2410ffb3d9293ded5c8100aa58dd2bd3f7795abd3bcd688f49b5c9318975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T02:21:18.839 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.436577226 +0000 UTC m=+0.016706461 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:21:18.839 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.536200742 +0000 UTC m=+0.116329956 container create 95d3e96e312b59fc71f4eefe48a17f80382c7df0aa596d8e60f6d3a8cb6bd65f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T02:21:18.839 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.763742196 +0000 UTC m=+0.343871430 container init 95d3e96e312b59fc71f4eefe48a17f80382c7df0aa596d8e60f6d3a8cb6bd65f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T02:21:18.839 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.767332681 +0000 UTC m=+0.347461898 container start 95d3e96e312b59fc71f4eefe48a17f80382c7df0aa596d8e60f6d3a8cb6bd65f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T02:21:19.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:19.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:18 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:18.577+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:19.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.838726258 +0000 UTC m=+0.418855501 container attach 95d3e96e312b59fc71f4eefe48a17f80382c7df0aa596d8e60f6d3a8cb6bd65f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T02:21:19.158 INFO:journalctl@ceph.osd.9.smithi016.stdout:Sep 21 02:21:18 smithi016 podman[105553]: 2024-09-21 02:21:18.966067681 +0000 UTC m=+0.546196903 container died 95d3e96e312b59fc71f4eefe48a17f80382c7df0aa596d8e60f6d3a8cb6bd65f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-9-deactivate, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T02:21:19.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:19.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:19.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:19.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:19.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:19.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:19 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:19.077+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:19.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.9.service' 2024-09-21T02:21:19.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:19.573 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-21T02:21:19.573 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-21T02:21:19.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.10 2024-09-21T02:21:19.899 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:19.899 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:19.899 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:19.899 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:19.899 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.606+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 systemd[1]: Stopping Ceph osd.10 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.897+0000 7f3ba011b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.897+0000 7f3ba011b640 -1 osd.10 104 *** Got signal Terminated *** 2024-09-21T02:21:19.900 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:19 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:19.897+0000 7f3ba011b640 -1 osd.10 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:20.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:20 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:20.043+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:21.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:20 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:20.636+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:21.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:21 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:21.088+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.597+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:22.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:21 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:21.598+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:22.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:22 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:22.110+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:23.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:23.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:22 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:22.579+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:23.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:23 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:23.124+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:23.922 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:23.923 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:23 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:23.615+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:24.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:24.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:24.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:24.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:24.415 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:24.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:24.416 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:24 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:24.140+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:24.937 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:58.952049+0000 front 2024-09-21T02:19:58.952064+0000 (oldest deadline 2024-09-21T02:20:24.152354+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:09.453139+0000 front 2024-09-21T02:20:09.453192+0000 (oldest deadline 2024-09-21T02:20:33.954082+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:13.954674+0000 front 2024-09-21T02:20:13.954579+0000 (oldest deadline 2024-09-21T02:20:39.854514+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-09-21T02:20:25.456851+0000 front 2024-09-21T02:20:25.457053+0000 (oldest deadline 2024-09-21T02:20:51.356659+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:32.957864+0000 front 2024-09-21T02:20:32.957880+0000 (oldest deadline 2024-09-21T02:21:00.558409+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:40.559078+0000 front 2024-09-21T02:20:40.559092+0000 (oldest deadline 2024-09-21T02:21:06.458803+0000) 2024-09-21T02:21:24.938 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:24 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10[57217]: 2024-09-21T02:21:24.578+0000 7f3b9c706640 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:51.660298+0000 front 2024-09-21T02:20:51.660448+0000 (oldest deadline 2024-09-21T02:21:16.960062+0000) 2024-09-21T02:21:25.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:25.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:25.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:25.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:25.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:25.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:25.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:25 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:25.165+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:25.715 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:25 smithi123 podman[66717]: 2024-09-21 02:21:25.44531745 +0000 UTC m=+5.667821514 container died 0247d23cf81c78e6422079a40df0b72c4be78635986fb6c5c582774a228bd150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:21:26.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:25 smithi123 podman[66717]: 2024-09-21 02:21:25.814221168 +0000 UTC m=+6.036725220 container cleanup 0247d23cf81c78e6422079a40df0b72c4be78635986fb6c5c582774a228bd150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T02:21:26.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:25 smithi123 bash[66717]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10 2024-09-21T02:21:26.424 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66741]: 2024-09-21 02:21:26.082034777 +0000 UTC m=+0.632517945 container remove 0247d23cf81c78e6422079a40df0b72c4be78635986fb6c5c582774a228bd150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:26.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:26 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:26.212+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:26.677 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.326680668 +0000 UTC m=+0.015702037 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:21:26.677 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.441349902 +0000 UTC m=+0.130371264 container create 72df48f4ca3a8b628427742889b73c26d9b05db3b7766262bd5fdacc63e04faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T02:21:26.677 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.677062286 +0000 UTC m=+0.366083655 container init 72df48f4ca3a8b628427742889b73c26d9b05db3b7766262bd5fdacc63e04faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:27.020 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.680497734 +0000 UTC m=+0.369519096 container start 72df48f4ca3a8b628427742889b73c26d9b05db3b7766262bd5fdacc63e04faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T02:21:27.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.760194022 +0000 UTC m=+0.449215406 container attach 72df48f4ca3a8b628427742889b73c26d9b05db3b7766262bd5fdacc63e04faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T02:21:27.021 INFO:journalctl@ceph.osd.10.smithi123.stdout:Sep 21 02:21:26 smithi123 podman[66834]: 2024-09-21 02:21:26.86990844 +0000 UTC m=+0.558929804 container died 72df48f4ca3a8b628427742889b73c26d9b05db3b7766262bd5fdacc63e04faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:27.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.10.service' 2024-09-21T02:21:27.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:27.497 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-21T02:21:27.497 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-21T02:21:27.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.11 2024-09-21T02:21:27.525 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:27.525 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:27.526 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:27.526 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:27.526 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:27.526 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:27.526 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:27.801 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 systemd[1]: Stopping Ceph osd.11 for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:28.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.800+0000 7f88966c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:28.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.800+0000 7f88966c9640 -1 osd.11 104 *** Got signal Terminated *** 2024-09-21T02:21:28.165 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:27 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:27.800+0000 7f88966c9640 -1 osd.11 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T02:21:28.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:28.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:28.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:28.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:28.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:28.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:28.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:28 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:28.252+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:29.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:29.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:29.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:29.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:29.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:29.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:29.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:29 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:29.236+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:30.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:30 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:30.191+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:31.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:31.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:31.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:31.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:31.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:31.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:31.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:31 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:31.240+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:32.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-09-21T02:19:59.833567+0000 front 2024-09-21T02:19:59.833421+0000 (oldest deadline 2024-09-21T02:20:22.133249+0000) 2024-09-21T02:21:32.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-09-21T02:20:06.535971+0000 front 2024-09-21T02:20:06.535843+0000 (oldest deadline 2024-09-21T02:20:31.835894+0000) 2024-09-21T02:21:32.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-09-21T02:20:14.737141+0000 front 2024-09-21T02:20:14.737024+0000 (oldest deadline 2024-09-21T02:20:38.837100+0000) 2024-09-21T02:21:32.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-09-21T02:20:30.340044+0000 front 2024-09-21T02:20:30.340210+0000 (oldest deadline 2024-09-21T02:20:56.240119+0000) 2024-09-21T02:21:32.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-09-21T02:20:41.541033+0000 front 2024-09-21T02:20:41.541060+0000 (oldest deadline 2024-09-21T02:21:04.441119+0000) 2024-09-21T02:21:32.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-09-21T02:20:50.842425+0000 front 2024-09-21T02:20:50.842497+0000 (oldest deadline 2024-09-21T02:21:16.142251+0000) 2024-09-21T02:21:32.666 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:32 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11[58804]: 2024-09-21T02:21:32.198+0000 7f888ecac640 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-09-21T02:21:00.144223+0000 front 2024-09-21T02:21:00.144142+0000 (oldest deadline 2024-09-21T02:21:23.643971+0000) 2024-09-21T02:21:33.718 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:33 smithi161 podman[68668]: 2024-09-21 02:21:33.457493928 +0000 UTC m=+5.757379544 container died 153de7da1b5c32838749a1681719335e760c4b162cdbac8e29bd5dfec53e852d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T02:21:33.993 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:33 smithi161 podman[68668]: 2024-09-21 02:21:33.800224282 +0000 UTC m=+6.100109903 container cleanup 153de7da1b5c32838749a1681719335e760c4b162cdbac8e29bd5dfec53e852d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T02:21:33.993 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:33 smithi161 bash[68668]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11 2024-09-21T02:21:34.402 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:33 smithi161 podman[68685]: 2024-09-21 02:21:33.992976175 +0000 UTC m=+0.531759526 container remove 153de7da1b5c32838749a1681719335e760c4b162cdbac8e29bd5dfec53e852d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:21:34.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.304450549 +0000 UTC m=+0.015741211 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T02:21:34.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.401955561 +0000 UTC m=+0.113246217 container create ad7c15c8af4ea1701c9bd226872f29317bb4f45a411a6cc9721751ad9b9e34ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T02:21:34.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.62904411 +0000 UTC m=+0.340334765 container init ad7c15c8af4ea1701c9bd226872f29317bb4f45a411a6cc9721751ad9b9e34ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T02:21:34.665 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.632374676 +0000 UTC m=+0.343665332 container start ad7c15c8af4ea1701c9bd226872f29317bb4f45a411a6cc9721751ad9b9e34ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T02:21:35.115 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.712452359 +0000 UTC m=+0.423743033 container attach ad7c15c8af4ea1701c9bd226872f29317bb4f45a411a6cc9721751ad9b9e34ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T02:21:35.115 INFO:journalctl@ceph.osd.11.smithi161.stdout:Sep 21 02:21:34 smithi161 podman[68771]: 2024-09-21 02:21:34.82017074 +0000 UTC m=+0.531461403 container died ad7c15c8af4ea1701c9bd226872f29317bb4f45a411a6cc9721751ad9b9e34ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-osd-11-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T02:21:35.415 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@osd.11.service' 2024-09-21T02:21:35.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:35.494 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-21T02:21:35.495 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-21T02:21:35.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.a 2024-09-21T02:21:35.820 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:35 smithi016 systemd[1]: Stopping Ceph mds.a for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:36.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: 2024-09-21T02:21:35.819+0000 7f8a8fdfd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:36.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:35 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a[74498]: 2024-09-21T02:21:35.819+0000 7f8a8fdfd640 -1 mds.a *** got signal Terminated *** 2024-09-21T02:21:37.845 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:37 smithi016 podman[105657]: 2024-09-21 02:21:37.488447678 +0000 UTC m=+1.793090976 container died 5cceb684e14078da0d95e15a17d7a715063ccbce961c4491ff853799f9111f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T02:21:38.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:37 smithi016 podman[105657]: 2024-09-21 02:21:37.969064997 +0000 UTC m=+2.273708288 container cleanup 5cceb684e14078da0d95e15a17d7a715063ccbce961c4491ff853799f9111f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True) 2024-09-21T02:21:38.158 INFO:journalctl@ceph.mds.a.smithi016.stdout:Sep 21 02:21:37 smithi016 bash[105657]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-a 2024-09-21T02:21:38.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.a.service' 2024-09-21T02:21:38.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:38.393 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-21T02:21:38.393 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-21T02:21:38.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.d 2024-09-21T02:21:38.723 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:38 smithi016 systemd[1]: Stopping Ceph mds.d for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:39.158 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:38 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-d[73373]: 2024-09-21T02:21:38.722+0000 7f9bdae81640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:39.158 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:38 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-d[73373]: 2024-09-21T02:21:38.722+0000 7f9bdae81640 -1 mds.d *** got signal Terminated *** 2024-09-21T02:21:43.149 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:42 smithi016 podman[105780]: 2024-09-21 02:21:42.735399271 +0000 UTC m=+4.119850795 container died 350fd734ec19ec0323ef132aba11f00bbcecfc7116215d7540489cc8a3d2745e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:43.408 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:43 smithi016 podman[105780]: 2024-09-21 02:21:43.223076191 +0000 UTC m=+4.607527714 container cleanup 350fd734ec19ec0323ef132aba11f00bbcecfc7116215d7540489cc8a3d2745e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T02:21:43.408 INFO:journalctl@ceph.mds.d.smithi016.stdout:Sep 21 02:21:43 smithi016 bash[105780]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-d 2024-09-21T02:21:43.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.d.service' 2024-09-21T02:21:43.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:43.691 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-21T02:21:43.691 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-21T02:21:43.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.g 2024-09-21T02:21:44.158 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:43 smithi016 systemd[1]: Stopping Ceph mds.g for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:44.158 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: 2024-09-21T02:21:44.018+0000 7f3f4d6ee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:44.158 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:44 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g[74119]: 2024-09-21T02:21:44.018+0000 7f3f4d6ee640 -1 mds.g *** got signal Terminated *** 2024-09-21T02:21:45.752 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:45 smithi016 podman[105901]: 2024-09-21 02:21:45.482587139 +0000 UTC m=+1.568872793 container died d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:46.111 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:45 smithi016 podman[105901]: 2024-09-21 02:21:45.843649819 +0000 UTC m=+1.929935458 container cleanup d2cd596694a259e9963a8555b35d46ae1ce624d34fe187daf040ad91a7bb1b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:21:46.111 INFO:journalctl@ceph.mds.g.smithi016.stdout:Sep 21 02:21:45 smithi016 bash[105901]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-g 2024-09-21T02:21:46.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.g.service' 2024-09-21T02:21:46.323 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:46.323 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-21T02:21:46.323 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-21T02:21:46.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.j 2024-09-21T02:21:46.649 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:46 smithi016 systemd[1]: Stopping Ceph mds.j for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:46.908 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j[72990]: 2024-09-21T02:21:46.647+0000 7ff3eae31640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:46.908 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:46 smithi016 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j[72990]: 2024-09-21T02:21:46.647+0000 7ff3eae31640 -1 mds.j *** got signal Terminated *** 2024-09-21T02:21:50.929 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:50 smithi016 podman[106025]: 2024-09-21 02:21:50.667985878 +0000 UTC m=+4.120746821 container died d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T02:21:51.199 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:51 smithi016 podman[106025]: 2024-09-21 02:21:51.006199738 +0000 UTC m=+4.458960678 container cleanup d6606fb804965a6f7cbff34a57a1813e79bf04742b934989aa071c585c27d0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T02:21:51.199 INFO:journalctl@ceph.mds.j.smithi016.stdout:Sep 21 02:21:51 smithi016 bash[106025]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-j 2024-09-21T02:21:51.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.j.service' 2024-09-21T02:21:51.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:51.459 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-21T02:21:51.459 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-21T02:21:51.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.b 2024-09-21T02:21:51.770 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:51 smithi123 systemd[1]: Stopping Ceph mds.b for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:52.270 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:21:51.771+0000 7f855a80d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:52.271 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:51 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b[62346]: 2024-09-21T02:21:51.771+0000 7f855a80d640 -1 mds.b *** got signal Terminated *** 2024-09-21T02:21:53.748 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:53 smithi123 podman[66937]: 2024-09-21 02:21:53.380372484 +0000 UTC m=+1.728648121 container died 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2) 2024-09-21T02:21:54.020 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:53 smithi123 podman[66937]: 2024-09-21 02:21:53.855496424 +0000 UTC m=+2.203772050 container cleanup 3f34c452239aea1f2e4aa5cccc19ef4d28530dafdf69fc2191df7111acd12349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T02:21:54.021 INFO:journalctl@ceph.mds.b.smithi123.stdout:Sep 21 02:21:53 smithi123 bash[66937]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-b 2024-09-21T02:21:54.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.b.service' 2024-09-21T02:21:54.325 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:54.326 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-21T02:21:54.326 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-21T02:21:54.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.e 2024-09-21T02:21:54.659 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:54 smithi123 systemd[1]: Stopping Ceph mds.e for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:55.020 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: 2024-09-21T02:21:54.658+0000 7f708797b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:55.020 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:54 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e[61803]: 2024-09-21T02:21:54.658+0000 7f708797b640 -1 mds.e *** got signal Terminated *** 2024-09-21T02:21:57.643 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:57 smithi123 podman[67059]: 2024-09-21 02:21:57.373672213 +0000 UTC m=+2.825641229 container died fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-09-21T02:21:58.020 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:57 smithi123 podman[67059]: 2024-09-21 02:21:57.733800449 +0000 UTC m=+3.185769474 container cleanup fcb7c60c4c90a7b8814cac96846f6732fb8b025caea5aefd608c45e071a42b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T02:21:58.021 INFO:journalctl@ceph.mds.e.smithi123.stdout:Sep 21 02:21:57 smithi123 bash[67059]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-e 2024-09-21T02:21:58.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.e.service' 2024-09-21T02:21:58.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:21:58.203 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-21T02:21:58.203 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-21T02:21:58.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.h 2024-09-21T02:21:58.520 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:21:58 smithi123 systemd[1]: Stopping Ceph mds.h for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:21:59.020 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:21:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: 2024-09-21T02:21:58.528+0000 7f06ff5e3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:21:59.021 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:21:58 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h[62078]: 2024-09-21T02:21:58.528+0000 7f06ff5e3640 -1 mds.h *** got signal Terminated *** 2024-09-21T02:22:01.646 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:22:01 smithi123 podman[67192]: 2024-09-21 02:22:01.375195711 +0000 UTC m=+2.956650696 container died 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T02:22:01.989 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:22:01 smithi123 podman[67192]: 2024-09-21 02:22:01.737596286 +0000 UTC m=+3.319051258 container cleanup 09b843ce12a5940c12a4557169832d8d754bbf22b5ab907d5fbefdd16777c137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:22:01.989 INFO:journalctl@ceph.mds.h.smithi123.stdout:Sep 21 02:22:01 smithi123 bash[67192]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-h 2024-09-21T02:22:02.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.h.service' 2024-09-21T02:22:02.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:02.152 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-21T02:22:02.152 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-21T02:22:02.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.k 2024-09-21T02:22:02.270 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:02 smithi123 systemd[1]: Stopping Ceph mds.k for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:22:02.770 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: 2024-09-21T02:22:02.456+0000 7f9d35d09640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:22:02.771 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:02 smithi123 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k[61526]: 2024-09-21T02:22:02.456+0000 7f9d35d09640 -1 mds.k *** got signal Terminated *** 2024-09-21T02:22:05.742 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:05 smithi123 podman[67315]: 2024-09-21 02:22:05.377696225 +0000 UTC m=+3.008051446 container died 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T02:22:06.000 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:05 smithi123 podman[67315]: 2024-09-21 02:22:05.824380484 +0000 UTC m=+3.454735705 container cleanup 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:22:06.000 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:05 smithi123 bash[67315]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k 2024-09-21T02:22:06.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.k.service' 2024-09-21T02:22:06.270 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:06 smithi123 podman[67329]: 2024-09-21 02:22:06.000086119 +0000 UTC m=+0.615580499 container remove 3c3f3738d16c0d31f992ef11e74fa266ddae5e77cb537d65d4e53248a0fa0ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T02:22:06.270 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:06 smithi123 systemd[1]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.k.service: Deactivated successfully. 2024-09-21T02:22:06.270 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:06 smithi123 systemd[1]: Stopped Ceph mds.k for 8372ca74-77bc-11ef-baf5-efdc52797490. 2024-09-21T02:22:06.270 INFO:journalctl@ceph.mds.k.smithi123.stdout:Sep 21 02:22:06 smithi123 systemd[1]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.k.service: Consumed 2.200s CPU time. 2024-09-21T02:22:06.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:06.282 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-21T02:22:06.282 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-21T02:22:06.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.c 2024-09-21T02:22:06.665 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:06 smithi161 systemd[1]: Stopping Ceph mds.c for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:22:06.928 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: 2024-09-21T02:22:06.665+0000 7fa543344640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:22:06.928 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:06 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c[63923]: 2024-09-21T02:22:06.665+0000 7fa543344640 -1 mds.c *** got signal Terminated *** 2024-09-21T02:22:09.665 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:09 smithi161 podman[68873]: 2024-09-21 02:22:09.385358098 +0000 UTC m=+2.913912559 container died 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T02:22:10.098 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:09 smithi161 podman[68873]: 2024-09-21 02:22:09.880941777 +0000 UTC m=+3.409496239 container cleanup 40d344ebc1c0975c2b0d0b2cd1a3f4ac010f8b0812ad715c6ec9388e49d13083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T02:22:10.099 INFO:journalctl@ceph.mds.c.smithi161.stdout:Sep 21 02:22:09 smithi161 bash[68873]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-c 2024-09-21T02:22:10.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.c.service' 2024-09-21T02:22:10.274 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:10.274 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-21T02:22:10.274 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-21T02:22:10.274 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.f 2024-09-21T02:22:10.665 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:10 smithi161 systemd[1]: Stopping Ceph mds.f for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:22:10.665 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: 2024-09-21T02:22:10.593+0000 7fe524a20640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:22:10.665 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:10 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f[63651]: 2024-09-21T02:22:10.593+0000 7fe524a20640 -1 mds.f *** got signal Terminated *** 2024-09-21T02:22:13.665 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:13 smithi161 podman[69006]: 2024-09-21 02:22:13.41022866 +0000 UTC m=+2.918314926 container died a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T02:22:14.044 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:13 smithi161 podman[69006]: 2024-09-21 02:22:13.776047957 +0000 UTC m=+3.284134216 container cleanup a15f11d79e22a9ab494582fba6822ee8c7e700b59b6b2e70cf39059bffde4ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T02:22:14.044 INFO:journalctl@ceph.mds.f.smithi161.stdout:Sep 21 02:22:13 smithi161 bash[69006]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-f 2024-09-21T02:22:14.206 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.f.service' 2024-09-21T02:22:14.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:14.240 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-21T02:22:14.241 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-21T02:22:14.241 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.i 2024-09-21T02:22:14.555 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:14 smithi161 systemd[1]: Stopping Ceph mds.i for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:22:14.915 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: 2024-09-21T02:22:14.554+0000 7fca75156640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:22:14.915 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:14 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i[64189]: 2024-09-21T02:22:14.554+0000 7fca75156640 -1 mds.i *** got signal Terminated *** 2024-09-21T02:22:17.665 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:17 smithi161 podman[69128]: 2024-09-21 02:22:17.39508626 +0000 UTC m=+2.933639096 container died 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T02:22:17.939 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:17 smithi161 podman[69128]: 2024-09-21 02:22:17.737464474 +0000 UTC m=+3.276017303 container cleanup 4a6b1fc6e4c3a9d84d56b375899907215acf92a89da88070ee9ea5efe4ac786b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731) 2024-09-21T02:22:17.939 INFO:journalctl@ceph.mds.i.smithi161.stdout:Sep 21 02:22:17 smithi161 bash[69128]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-i 2024-09-21T02:22:18.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.i.service' 2024-09-21T02:22:18.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:18.192 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-21T02:22:18.192 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-21T02:22:18.192 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.l 2024-09-21T02:22:18.516 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:18 smithi161 systemd[1]: Stopping Ceph mds.l for 8372ca74-77bc-11ef-baf5-efdc52797490... 2024-09-21T02:22:18.915 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: 2024-09-21T02:22:18.515+0000 7fb7cdd93640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T02:22:18.915 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:18 smithi161 ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l[63368]: 2024-09-21T02:22:18.515+0000 7fb7cdd93640 -1 mds.l *** got signal Terminated *** 2024-09-21T02:22:21.665 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:21 smithi161 podman[69250]: 2024-09-21 02:22:21.394926581 +0000 UTC m=+2.989330557 container died 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T02:22:22.033 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:21 smithi161 podman[69250]: 2024-09-21 02:22:21.799138005 +0000 UTC m=+3.393541973 container cleanup 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T02:22:22.034 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:21 smithi161 bash[69250]: ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l 2024-09-21T02:22:22.284 INFO:journalctl@ceph.mds.l.smithi161.stdout:Sep 21 02:22:22 smithi161 podman[69262]: 2024-09-21 02:22:22.033215989 +0000 UTC m=+0.639003761 container remove 5345a615119808cb66200d638f90cb5e0e633ff975674544fac3f2ca50acc93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-8372ca74-77bc-11ef-baf5-efdc52797490-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T02:22:22.286 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8372ca74-77bc-11ef-baf5-efdc52797490@mds.l.service' 2024-09-21T02:22:22.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T02:22:22.319 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-21T02:22:22.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T02:22:22.534 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:22:25.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T02:22:25.428 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:22:27.887 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T02:22:28.089 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:22:30.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:22:30.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:22:30.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T02:22:30.588 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T02:22:30.590 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi016/crash 2024-09-21T02:22:30.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash -- . 2024-09-21T02:22:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T02:22:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-09-21T02:22:30.625 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi123/crash 2024-09-21T02:22:30.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash -- . 2024-09-21T02:22:30.734 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T02:22:30.734 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-09-21T02:22:30.736 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi161/crash 2024-09-21T02:22:30.738 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash -- . 2024-09-21T02:22:30.774 INFO:teuthology.orchestra.run.smithi161.stderr:tar: /var/lib/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T02:22:30.774 INFO:teuthology.orchestra.run.smithi161.stderr:tar: Error is not recoverable: exiting now 2024-09-21T02:22:30.775 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T02:22:30.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-21T02:22:30.883 INFO:tasks.cephadm:Compressing logs... 2024-09-21T02:22:30.884 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T02:22:30.886 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T02:22:30.888 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T02:22:30.915 INFO:teuthology.orchestra.run.smithi161.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T02:22:30.915 INFO:teuthology.orchestra.run.smithi161.stderr:: No such file or directory 2024-09-21T02:22:30.915 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.i.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log: /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.11.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.l.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.8.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.z.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.c.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.f.log 2024-09-21T02:22:30.916 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.10.log 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:: No such file or directory 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T02:22:30.918 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.b.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.k.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log: 81.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T02:22:30.920 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log 2024-09-21T02:22:30.921 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.y.log: 90.2% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T02:22:30.926 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.f.log: /var/log/ceph/cephadm.log: 81.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T02:22:30.928 INFO:teuthology.orchestra.run.smithi161.stderr: 86.7% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T02:22:30.928 INFO:teuthology.orchestra.run.smithi123.stderr: 92.8% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T02:22:30.929 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.h.log 2024-09-21T02:22:30.929 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-21T02:22:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T02:22:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-21T02:22:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T02:22:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.9.log 2024-09-21T02:22:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T02:22:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T02:22:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.a.log 2024-09-21T02:22:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T02:22:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-21T02:22:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.g.log 2024-09-21T02:22:30.933 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.e.log 2024-09-21T02:22:30.933 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.b.log: 86.7% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T02:22:30.933 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T02:22:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T02:22:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log 2024-09-21T02:22:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.d.log 2024-09-21T02:22:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-21T02:22:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.j.log 2024-09-21T02:22:30.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T02:22:30.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-09-21T02:22:30.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T02:22:30.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: 86.7% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T02:22:30.938 INFO:teuthology.orchestra.run.smithi016.stderr: 95.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.d.log.gz 2024-09-21T02:22:30.976 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.e.log: /var/log/ceph/cephadm.log: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T02:22:30.987 INFO:teuthology.orchestra.run.smithi016.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-21T02:22:30.995 INFO:teuthology.orchestra.run.smithi123.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T02:22:31.022 INFO:teuthology.orchestra.run.smithi123.stderr: 94.6% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.k.log.gz 2024-09-21T02:22:31.038 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.h.log.gz 2024-09-21T02:22:31.049 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T02:22:31.082 INFO:teuthology.orchestra.run.smithi161.stderr: 93.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.c.log.gz 2024-09-21T02:22:31.126 INFO:teuthology.orchestra.run.smithi161.stderr: 93.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.i.log.gz 2024-09-21T02:22:31.136 INFO:teuthology.orchestra.run.smithi123.stderr: 93.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.e.log.gz 2024-09-21T02:22:31.140 INFO:teuthology.orchestra.run.smithi161.stderr: 92.8% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.z.log.gz 2024-09-21T02:22:31.143 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.f.log.gz 2024-09-21T02:22:31.166 INFO:teuthology.orchestra.run.smithi161.stderr: 90.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T02:22:31.232 INFO:teuthology.orchestra.run.smithi016.stderr: 95.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.j.log.gz 2024-09-21T02:22:31.236 INFO:teuthology.orchestra.run.smithi161.stderr: 94.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.l.log.gz 2024-09-21T02:22:31.327 INFO:teuthology.orchestra.run.smithi161.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T02:22:31.350 INFO:teuthology.orchestra.run.smithi016.stderr: 90.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T02:22:31.371 INFO:teuthology.orchestra.run.smithi123.stderr: 91.7% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-21T02:22:31.383 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T02:22:31.622 INFO:teuthology.orchestra.run.smithi016.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T02:22:31.649 INFO:teuthology.orchestra.run.smithi016.stderr: 84.2% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T02:22:31.706 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.a.log.gz 2024-09-21T02:22:31.772 INFO:teuthology.orchestra.run.smithi161.stderr: 91.8% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-21T02:22:31.900 INFO:teuthology.orchestra.run.smithi123.stderr: 93.8% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.b.log.gz 2024-09-21T02:22:32.273 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T02:22:32.980 INFO:teuthology.orchestra.run.smithi016.stderr: 94.4% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mds.g.log.gz 2024-09-21T02:22:40.633 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-21T02:23:04.846 INFO:teuthology.orchestra.run.smithi123.stderr: 91.2% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.10.log.gz 2024-09-21T02:23:18.006 INFO:teuthology.orchestra.run.smithi123.stderr: 91.2% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T02:23:21.427 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.11.log.gz 2024-09-21T02:23:28.308 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.8.log.gz 2024-09-21T02:23:28.658 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T02:23:35.170 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T02:23:43.994 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T02:23:43.996 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-21T02:23:43.996 INFO:teuthology.orchestra.run.smithi123.stderr:real 1m13.092s 2024-09-21T02:23:43.996 INFO:teuthology.orchestra.run.smithi123.stderr:user 3m26.152s 2024-09-21T02:23:43.996 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m6.582s 2024-09-21T02:24:05.016 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T02:24:05.018 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-09-21T02:24:05.018 INFO:teuthology.orchestra.run.smithi161.stderr:real 1m34.116s 2024-09-21T02:24:05.018 INFO:teuthology.orchestra.run.smithi161.stderr:user 3m37.446s 2024-09-21T02:24:05.018 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m7.252s 2024-09-21T02:27:06.898 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T02:27:58.143 INFO:teuthology.orchestra.run.smithi016.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T02:28:52.846 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.9.log.gz 2024-09-21T02:29:25.798 INFO:teuthology.orchestra.run.smithi016.stderr: 90.9% -- replaced with /var/log/ceph/8372ca74-77bc-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T02:29:58.029 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-09-21T02:29:58.031 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-21T02:29:58.031 INFO:teuthology.orchestra.run.smithi016.stderr:real 7m27.131s 2024-09-21T02:29:58.031 INFO:teuthology.orchestra.run.smithi016.stderr:user 5m23.830s 2024-09-21T02:29:58.031 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m12.375s 2024-09-21T02:29:58.032 INFO:tasks.cephadm:Archiving logs... 2024-09-21T02:29:58.033 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi016/log 2024-09-21T02:29:58.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T02:30:47.388 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi123/log 2024-09-21T02:30:47.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T02:31:21.815 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi161/log 2024-09-21T02:31:21.816 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T02:31:58.516 INFO:tasks.cephadm:Removing cluster... 2024-09-21T02:31:58.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force 2024-09-21T02:31:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:31:59.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force 2024-09-21T02:31:59.532 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:32:00.168 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8372ca74-77bc-11ef-baf5-efdc52797490 --force 2024-09-21T02:32:00.373 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: 8372ca74-77bc-11ef-baf5-efdc52797490 2024-09-21T02:32:00.968 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T02:32:00.968 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T02:32:00.988 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T02:32:01.004 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T02:32:01.020 INFO:tasks.cephadm:Teardown complete 2024-09-21T02:32:01.020 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T02:32:01.030 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T02:32:01.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T02:32:01.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T02:32:01.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T02:32:01.166 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T02:32:01.176 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T02:32:01.177 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:32:01.179 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:32:01.182 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:32:01.194 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:32:01.195 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:32:01.196 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 -6592ns[ +23us] +/- 40ms 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -663us[ -634us] +/- 79ms 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +637us[ +667us] +/- 70ms 2024-09-21T02:32:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:32:01.201 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:32:01.201 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-21T02:32:01.201 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +53us[ +49us] +/- 40ms 2024-09-21T02:32:01.201 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 189 -576us[ -588us] +/- 78ms 2024-09-21T02:32:01.201 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +677us[ +673us] +/- 70ms 2024-09-21T02:32:01.202 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:32:01.223 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:32:01.223 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-09-21T02:32:01.223 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 +9929ns[ +46us] +/- 40ms 2024-09-21T02:32:01.223 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 58 -662us[ -662us] +/- 80ms 2024-09-21T02:32:01.223 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +627us[ +627us] +/- 70ms 2024-09-21T02:32:01.224 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:32:01.224 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T02:32:01.239 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T02:32:01.239 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T02:32:01.260 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T02:32:01.379 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T02:32:01.458 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T02:32:01.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:32:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:01.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:32:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:01.603 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T02:32:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:01.630 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T02:32:01.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T02:32:01.677 INFO:teuthology.task.internal:Duration was 2556.919518 seconds 2024-09-21T02:32:01.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T02:32:01.711 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T02:32:01.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T02:32:01.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T02:32:01.716 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T02:32:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:32:01.786 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:32:01.790 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:32:02.227 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T02:32:02.228 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-21T02:32:02.228 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T02:32:02.252 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-09-21T02:32:02.252 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T02:32:02.279 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-09-21T02:32:02.280 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T02:32:02.313 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T02:32:02.313 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T02:32:02.316 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T02:32:02.322 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T02:32:02.380 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T02:32:02.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T02:32:02.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T02:32:02.391 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T02:32:03.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T02:32:03.210 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T02:32:03.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T02:32:03.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T02:32:03.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T02:32:03.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T02:32:03.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.326 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.348 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-09-21T02:32:03.352 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-21T02:32:03.381 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-09-21T02:32:03.397 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:03.425 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:03.440 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:32:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:32:03.456 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T02:32:03.466 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T02:32:03.467 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi016 2024-09-21T02:32:03.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T02:32:03.520 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi123 2024-09-21T02:32:03.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T02:32:03.555 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913403/remote/smithi161 2024-09-21T02:32:03.556 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T02:32:03.593 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T02:32:03.593 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T02:32:03.596 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T02:32:03.598 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T02:32:03.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T02:32:03.656 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T02:32:03.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T02:32:03.720 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T02:32:03.720 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T02:32:03.722 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T02:32:03.725 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T02:32:03.736 INFO:teuthology.orchestra.run.smithi016.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 02:32 /home/ubuntu/cephtest 2024-09-21T02:32:03.737 INFO:teuthology.orchestra.run.smithi123.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 02:32 /home/ubuntu/cephtest 2024-09-21T02:32:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 02:32 /home/ubuntu/cephtest 2024-09-21T02:32:03.741 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T02:32:03.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T02:32:03.823 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iogen}} duration: 2556.919517993927 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-21T02:32:03.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T02:32:03.904 INFO:teuthology.run:pass